Language models can read the same sentence and produce completely different lists of numbers. Those numbers locate the sentence in each model’s internal space, which has many dimensions and no direct English interpretation.
This experiment asks:
If we learn how one model arranges ideas, can we predict where the other model will place a sentence it has never seen before?
For three models and two fixed prompt sets, the answer is yes in all six directions. The strength of the result depends on which model is the source and which is the target.
If “representation,” “FUV,” or “permutation baseline” is unfamiliar, the plain-language glossary explains each term with a small example. The equations are optional.
First, imagine a room full of sentences
The real representations have 896 or 2,048 coordinates. The illustration below compresses that idea into two dimensions. Each dot is one prompt. Matching labels mean both models read the same prompt.
A two-dimensional illustration
Select a prompt to see where each model places the same sentence.
Prompts that carry related information can preserve enough shared structure across these different arrangements for a simple linear translator to work between the spaces.
Learn on some prompts, test on different ones
The translator learns from 48 paired prompts. It is then frozen and tested on 16 held-out prompts that could not influence the fit or its regularization.
The held-out prompts distinguish reconstruction from memorization: the score measures how well the fitted translator predicts representations for prompts it has not seen.
The score is calculated only on the orange held-out quarter.
What would failure look like?
To interpret a low score, we compare it with a version where the relationship has been deliberately broken.
Press shuffle below. The models keep their real internal geometry, but Qwen’s representation of “volcano” may be paired with TinyLlama’s representation of “recipe.” The prompt-to-prompt relationship disappears.
With the correct pairings, the translator scores 0.818 FUV on TinyLlama’s held-out representations. After prompt identities are broken, the typical score is 1.103. The difference between those quantities measures the structure associated with correct prompt alignment.
Show me the math
- 0 means perfect reconstruction.
- 1 means the translator is no better than always guessing the average representation.
- Above 1 means it is worse than that simple guess.
The study's primary quantity is real FUV minus shuffled FUV. Negative is good: the true prompt alignment explains structure that disappears when the identities are broken.
Start with one pair
A single train/test split would provide weak evidence. Before inspecting the corresponding results, each extension froze 20 split seeds, two null baselines, the regularization grid, and the reporting rules in a committed preregistration.
The chart shows every split. Each line runs from the real score to the median shuffled score. Farther left is better.
Median real FUV0.818
Median advantage over shuffle0.285
Splits won20 of 20
Every direction beat the shuffled baseline on all 20 splits. Use the buttons to inspect the six maps. The chart keeps every split visible alongside the summary statistics.
These numbers describe linear translatability. They do not compare the models’ intelligence or overall quality. Each direction has different target variance and representation geometry, so cleaner translation in one direction does not imply that one model is generally better.
The three-model triangle
One pair could make directionality look like a general rule. Adding SmolLM2 lets us check all three pairings. Choose an edge below. Each number is the median advantage over shuffled prompt matching; larger is better in this display.
Qwen ↔ TinyLlama
Qwen → TinyLlama0.285
TinyLlama → Qwen0.454
TinyLlama was the stronger source on every paired split.
Qwen and SmolLM2 were close to symmetric: their paired sensitivity range crossed zero. Both pairings involving TinyLlama were asymmetric: TinyLlama → Qwen and TinyLlama → SmolLM2 had the larger adjusted advantage on every paired split.
Model size alone does not explain the pattern. TinyLlama and SmolLM2 both have 2,048-coordinate representations, yet their two directions differ substantially. A later experiment can test whether properties of the target representation, including how its variation is distributed, account for the differences. Three checkpoints do not test that explanation.
Test on a second prompt set
The first triangle used one deliberately constructed set of 64 prompts. Before creating any new model vectors, the replication froze a second set of 64 newly written prompts and the analysis rules that would be applied to them. Switch between the sets below. Each bar is the median advantage over shuffled prompt matching; longer is better.
First 64 prompts
All six directions beat shuffled prompt matching on all 20 splits.
The second set produced the same three descriptive findings. Every direction again beat its matched shuffled baseline on all 20 splits. TinyLlama again had the larger advantage as the source in both pairings on every paired split. The Qwen-SmolLM2 sensitivity range again crossed zero.
Scope of the result
Across two fixed 64-prompt fixtures, the experiment establishes a stable linear association among three frozen checkpoints. The first Qwen-SmolLM2 study was rerun on a second machine, and all 2,165 numeric values matched within 1e-12. Each of the three second-fixture result files was rerun in a separate process; after excluding its creation timestamp, every remaining field was exactly identical. A separate read-only audit recomputed the reported medians, sign counts, quantiles, and permutation p-values.
The study has not yet tested the larger 16-model theory. Both prompt sets were deliberately written for this experiment. Neither was sampled from a defined prompt population. The result also does not establish a RankMe relationship, identify a causal mechanism, or rank model quality. Because the 20 splits overlap within each fixture, they serve as sensitivity checks rather than independent experiments.
Inspect the repository · Read the replication design · Read the verified results · Use the glossary