This page explains the terms used in Diode Spectrum, an experiment about translating among language models’ internal representations. Each entry starts with the practical idea. The optional sections contain the notation.
FUV
FUV stands for fraction of unexplained variance. It measures how much of the target’s variation the reconstructed vectors failed to capture. Lower is better.
Tiny example: imagine predicting four dots on a page. FUV compares your total prediction error with the error from placing every prediction at the center of the four dots. Small misses produce a low FUV.
Move the FUV score
Drag the control. The only rule you need is: lower is better.
Better than always guessing the average. Lower is better.
FUV = 0 · nothing left unexplained
Every held-out target is reconstructed exactly.
FUV = 1 · equal to the average guess
The map's total squared error matches the error from predicting the held-out target mean for every prompt.
FUV > 1 · worse than the average guess
The map has more total squared error than that mean-prediction reference.
A low FUV establishes reconstructability under the chosen map, prompts, layer, pooling rule, and split. It says nothing by itself about overall model quality, interchangeable behavior, shared mechanisms, or semantic equivalence.
Show the FUV formula
For held-out target matrix $Y$ and reconstruction $\hat{Y}$:
$$ \operatorname{FUV}(\hat{Y},Y) = \frac{\sum_{i,j}(\hat{Y}_{ij}-Y_{ij})^2} {\sum_{i,j}(Y_{ij}-\bar{Y}_{\cdot j})^2}. $$
The top is reconstruction error. The bottom is the held-out target's variation around its coordinate-wise mean.
Adjusted FUV gap
The adjusted FUV gap asks how much correct prompt matching helps after accounting for a deliberately broken version of the experiment.
Tiny example: suppose the real map scores 0.80 FUV and shuffled prompt pairings usually score 1.10. The adjusted gap is 0.80 minus 1.10, or -0.30. The negative value says the correctly paired prompts reconstructed better.
A more negative gap means a larger advantage over the row-permutation baseline. The two directions can still differ in dimensions and geometry, so baseline adjustment does not make them fully equivalent.
Show the calculation
Artifact hash and manifest
An artifact hash works like a fingerprint for a file. A manifest is the packing list that says which fingerprints and run details should be present.
Tiny example: if a hidden-state file has a recorded SHA-256 hash, changing one number changes the hash. A verifier can then detect that the file is no longer the frozen input named in the manifest.
Diode’s manifests also record items such as model identifiers, prompt rows, commands, and splits. Matching hashes establish that the bytes are identical. They do not establish that the research question or experimental design was sound.
Claim boundary
A claim boundary is the line between what a result supports and what remains unknown.
Tiny example: a frozen panel of 16 checkpoints supports a statement about those checkpoints and prompt batteries. It does not estimate a law over all language models or a defined population of prompts.
Diode’s current result is a panel-level association about 16 frozen checkpoints across 12 families. It does not identify a causal mechanism, establish semantic equivalence, rank the models’ quality, or support downstream utility claims. Recording this boundary in the design and result artifacts lets reviewers compare later claims with the original scope.
Directionality and asymmetry
Directionality means that translating from model A to model B is a different task from translating B to A. Asymmetry measures how differently those two tasks perform after baseline adjustment.
Tiny example: if Qwen → TinyLlama has an adjusted gap of -0.29 and TinyLlama → Qwen has -0.45, the second direction has the larger advantage over its own baseline because more negative is better.
Reversing a map changes the input width, output width, target variance, and geometry, so each direction needs its own fit and score. Asymmetry describes linear reconstructability in this setup. It carries no general model-quality interpretation.
Show the asymmetry calculation
The sign only has meaning after the forward direction has been declared.
Gaussian-source floor
The Gaussian-source floor reruns the analysis with random numbers in place of a real model’s source representations. It shows what the same pipeline can achieve without prompt-aligned model information.
Tiny example: replace Qwen’s 64 prompt vectors with a same-sized table of Gaussian random draws, select the ridge penalty again, fit again, and score the same held-out target prompts.
This check exposes gains caused by sample size and dimensionality alone. The random table does not preserve the real model’s spectrum or geometry. The three-model bidirectional study used this floor as a secondary compatibility check and row permutation as its primary baseline.
The 16-model panel makes the floor its primary H3 statistic: for every directed pair, the same ridge pipeline runs with Gaussian source tensors of the same shape from 20 matched seeds, and the pair’s floor is the mean over those seeds. Observed minus Gaussian is negative when real-source reconstruction beat the matched random floor.
Layer
A layer is one stage in the model’s sequence of computations. The representation changes as a prompt moves through those stages.
Tiny example: the vector for “volcano” after layer 4 can differ from its vector after the final layer, even though both came from the same prompt and model.
Diode’s reported pilot and sensitivity results use the last layer of each frozen model. Evidence from that layer does not automatically apply to earlier or middle layers.
Linear map
A linear map is a single coordinate-conversion rule. It stretches, shrinks, mixes, and offsets the source numbers to predict target numbers.
Tiny example: a simple map might predict one target coordinate as twice the first source coordinate plus half the second. The same learned rule is used for every prompt.
Linearity is intentionally restrictive. Success on held-out prompts shows that some cross-model structure is accessible through a simple global transformation. A failed linear map would still leave open the possibility of a more complicated nonlinear relationship.
Show the linear-map notation
$$\hat{Y}=XW+b.$$
$X$ contains source representations, $W$ is the learned conversion matrix, and $b$ is an offset.
Masked-mean pooling
Masked-mean pooling turns all of a prompt’s token vectors into one prompt vector by averaging only the real token positions.
Tiny example: if “a red kite” becomes three tokens plus two padding slots, the average uses the three token vectors and ignores the padding.
The attention mask marks which positions count. Pooling is part of the measurement: last-token pooling, unmasked averaging, or another rule can produce different vectors and a different result.
Preregistration
A preregistration is a time-stamped plan written before the new result is inspected. It limits the researcher’s ability to keep trying analyses until one looks favorable.
Tiny example: before running a new model pair, the plan fixes the prompt split seeds, ridge choices, baselines, summary statistics, and claim boundary.
Diode binds the frozen plan to later artifacts by hash. Preregistration does not guarantee a strong hypothesis or bug-free code, so tests, independent verification, and restrained interpretation still matter.
RankMe and effective rank
Effective rank describes how many directions in a representation space carry meaningful variation. RankMe is the particular effective-rank measure used by Diode.
Tiny example: a cloud of points lying almost entirely along one line has RankMe near 1. A cloud spread evenly across several directions has a larger RankMe.
RankMe summarizes how concentrated the variation is across directions; it should not be read as the number of concepts a model knows. Diode computes RankMe once per model on the full 4,000-row centered tensor, keeping singular values above 1e-15 times the largest one.
Show the RankMe formula
Center the representation matrix and compute its singular values. Normalize them so their shares $p_k$ add to one, then calculate:
$$ \operatorname{RankMe} = \exp\left(-\sum_k p_k\log p_k\right). $$
Representation or hidden state
A hidden state is a list of numbers the model carries while processing a token. A representation is the vector the experiment chooses to measure from those hidden states.
Tiny example: the model produces one hidden-state vector for each token in “the moon rises.” Diode averages the last-layer token vectors into one representation for the whole prompt.
Individual coordinates do not come with fixed English labels. Diode studies the pattern across prompts: whether one model’s vectors can reconstruct another model’s vectors for prompts excluded from fitting.
Ridge regression
Ridge regression learns a linear map while discouraging extreme weights. The penalty makes the fit more stable when there are many coordinates and few training prompts.
Tiny example: two maps may fit the 48 training prompts equally well, but one uses enormous positive and negative weights. Ridge usually prefers the map with smaller weights because it is less likely to react strongly to a new prompt.
Diode chooses the penalty strength using folds inside the training set. The held-out prompts cannot influence that choice or the fitted weights.
Show the ridge objective
$$ \lVert XW-Y\rVert_F^2+\lambda\lVert W\rVert_F^2. $$
The first term measures training error. The second penalizes large weights. $\lambda$ controls the tradeoff.
Row-permutation baseline
The row-permutation baseline breaks prompt matching while keeping the real source vectors themselves unchanged.
Tiny example: Qwen’s vector for “volcano” might be paired with TinyLlama’s vector for “recipe.” Every source row still exists, but it points to the wrong target prompt.
This preserves the source tensor’s exact scale, spectrum, and internal geometry while removing the intended row correspondence. Each shuffled version repeats ridge selection and fitting. Beating it supports a prompt-aligned association; it does not explain which information creates the association.
Split and sensitivity range
A split decides which prompts teach the map and which prompts test it. A sensitivity range shows how much the result changes across several such divisions.
Tiny example: one split may hold out “volcano,” while another trains on it and holds out “recipe.” Diode summarizes the scores across 20 frozen splits with a median and empirical 2.5th-to-97.5th percentile range.
The same prompts appear in many splits, so the 20 results are statistically dependent. They show sensitivity to different train/test divisions, not evidence from 20 independent experiments. The reported range therefore does not estimate performance on new prompt collections.
Train and held-out prompts
Training prompts are the examples allowed to shape the map. Held-out prompts are sealed off until the final score is calculated.
Tiny example: in each reported sensitivity split, 48 paired prompts select and fit the ridge map. The remaining 16 prompts test whether it works on rows it never fitted.
This separation tests extension beyond the training rows. Repeating the complete analysis on a second fixed 64-prompt fixture tests whether the finding recurs on new rows. Both fixtures were deliberately constructed, so the result does not estimate performance across a defined population of everyday prompts.
Frozen panel and effective sample unit
A frozen panel is the fixed set of checkpoints and the fixed numerical design chosen before results are inspected. The effective sample unit is the smallest set of observations that can be treated as independent.
Tiny example: Diode’s panel freezes 16 models across 12 families, 4 prompt batteries, 10 split seeds, the ridge grid, and the reporting rules before any tensor is scored. The 240 directed pairs are not 240 independent experiments: every pair shares one of the 16 tensors as source or target, and all pairs use the same batteries and splits. The panel of 16 models is the unit that carries the evidence.
Family-wild null
The family-wild null is a Freedman-Lane residual-randomization test that respects the panel’s shared model families. It first fits the controls-only model, then applies independent random signs at the source-family and target-family levels to its residuals; each directed pair receives the product of its source and target family signs.
Tiny example: all pair rows with a qwen source share one source-family sign, and all rows with a phi target share one target-family sign. A qwen-to-phi row receives the product of those signs. The randomized residuals are added back to the controls-only fitted values, the full model is refit, and a draw whose absolute target-RankMe coefficient is at least as large as the observed one is an exceedance.
Diode’s H1 uses 50,000 draws at seed 314159 and reports an add-one p-value. The observed coefficient beat every draw, so the p-value is 1/50,001 = 0.0000199996 with zero exceedances.
Leave-one-family-out
Leave-one-family-out (LOFO) refits the full model with one model family excluded, to show the estimate does not depend on any single family.
Tiny example: dropping all phi models and refitting gives a new target-RankMe coefficient; doing this for each of the 12 families yields 12 coefficients. Diode’s H1 threshold requires at least 80% of these to be positive; all 12 are.
Add-one p-value
An add-one p-value counts the null draws at or below the observed statistic, plus one for the observed value itself, divided by one plus the number of draws.
Tiny example: with 20 matched Gaussian seeds, Diode’s H3 p-value is (1 + number of seeds whose panel mean is at or below the observed mean) / 21. The possible values are coarse steps of 1/21; the observed p = 1/21 ≈ 0.047619 is the smallest nonzero step.
Oracle floor
The oracle floor is the best error a mechanical procedure can achieve in a diagnostic, used as a lower bound for the real pipeline’s score.
Tiny example: Diode’s H6 truncates the target to its top-k training components, keeping at least 90% of training variance, and records the FUV of that truncation as the floor. The scored ridge FUV must not fall below the floor by more than 1e-10. This checks the pipeline; it is not a significance test.
Sentinel pair
A sentinel pair is a small fixed set of directed pairs used for cheap deterministic controls.
Tiny example: Diode runs its H5/H6 mechanical checks on six exact ordered pairs (both directions of three pairs) at one frozen battery, split seed, and training size. All six pass both checks.
Nested training sizes
Nested training sizes are training sets where each larger size contains all smaller ones, so size comparisons are not confounded by different row content.
Tiny example: within one split seed and battery, the n = 64 training rows are the first 64 rows of the same ordered 800-row pool, and n = 128 is those 64 plus the next 64. Diode sweeps n = 64, 128, 256, 512, 800 this way, with the same 200 held-out rows throughout.
For the full study narrative and current results, return to Diode Spectrum. Other public research and production work is listed on the projects page.