Learning to Read Out · Research explorer
A language model uses learned output weights, called its readout, to turn internal representations into scores for possible next tokens: words or pieces of words. These weights change during training. I developed trajectory crosscoding to track patterns in them, called features, across saved training checkpoints.
The figures below follow these features and test how they relate to predictions. The central question is whether information becomes detectable inside a model before its readout can express it reliably. That distinction matters when we use a model’s answers to judge what it has learned.
Inside the readout
Hover or tap a curve to inspect it.
Choose a model and move the training slider or replay the checkpoints. Hover, tap, or focus the chart and use the arrow keys to inspect a curve. In the feature examples, select Early, Peak, or Final to highlight that checkpoint and its tokens.
Feature curves show decoder norms divided by each feature’s own maximum. Their height shows change over time, not importance relative to other features. The trajectory explorer samples 1,200 active features per model; the original figure shows the full populations. Straight lines join measured checkpoints, and the training axis is logarithmic. Each model has its own training schedule.
Token lists follow the paper’s ordering, with display whitespace trimmed. Other plots retain the values and baselines of the paper figures; exact values are provided where tabular summaries are useful. Colours and marker shapes identify each condition.
02 · Individual features
To help interpret a feature, we inspect the tokens most associated with it at different stages of training. Select Early, Peak, or Final to see how the list changes alongside the curve. Labels such as “Chemistry” summarize these token lists and offer a qualitative interpretation.
Pythia-1B · Feature 1,227 · Peak at step 1,000
Peak · step 1,000 · relative norm 1.000
Pythia-1B · Feature 15,921 · Peak at step 2,000
Peak · step 2,000 · relative norm 1.000
Pythia-1B · Feature 2,228 · Peak at step 116,000
Peak · step 116,000 · relative norm 1.000
Three selected examples from the Pythia-1B dictionary of 24,576 features. Select Early, Peak, or Final to connect a point on the curve with its token list.
03 · Interventions
The agreement test asks whether the readout prefers a verb that matches the subject: for example, “The keys are” over “The keys is”. Removing eight selected feature contributions from the reconstructed readout reduced accuracy from 90.0% to 48.7%. Removing features chosen as matched controls left accuracy at 90.0%.
The panels compare three interventions: remove the selected contributions from the readout, keep only those contributions, or remove the corresponding directions from the internal representations. These tests help distinguish a feature’s association with a task from its effect on the answer scores.
| Task | Condition | Change in accuracy · percentage points |
|---|---|---|
| Subject–verb agreement | Eight selected directions | -41.3 |
| Subject–verb agreement | Norm and rate control | 0.0 |
| Subject–verb agreement | Sign control | 3.1 |
| Numeric comparison | Eight selected directions | -42.8 |
| Numeric comparison | Norm and rate control | 0.0 |
| Numeric comparison | Sign control | -1.2 |
| Relational facts* | Eight selected directions | -7.7 |
| Relational facts* | Norm and rate control | 0.0 |
| Relational facts* | Sign control | 0.0 |
| Indirect object identification | Eight selected directions | -3.5 |
| Indirect object identification | Norm and rate control | 0.0 |
| Indirect object identification | Sign control | -0.5 |
| Task | Condition | Accuracy · % |
|---|---|---|
| Subject–verb agreement | Eight selected directions | 93.8 |
| Subject–verb agreement | Reconstructed baseline | 90.0 |
| Subject–verb agreement | Sign control | 46.3 |
| Numeric comparison | Eight selected directions | 89.4 |
| Numeric comparison | Reconstructed baseline | 51.2 |
| Numeric comparison | Sign control | 50.7 |
| Relational facts* | Eight selected directions | 92.3 |
| Relational facts* | Reconstructed baseline | 23.1 |
| Relational facts* | Sign control | 30.8 |
| Indirect object identification | Eight selected directions | 48.1 |
| Indirect object identification | Reconstructed baseline | 42.9 |
| Indirect object identification | Sign control | 42.0 |
| Task | Condition | Accuracy · % |
|---|---|---|
| Subject–verb agreement | Native baseline | 78.1 |
| Subject–verb agreement | After projection | 50.0 |
| Numeric comparison | Native baseline | 60.1 |
| Numeric comparison | After projection | 49.7 |
| Relational facts* | Native baseline | 30.8 |
| Relational facts* | After projection | 15.4 |
| Indirect object identification | Native baseline | 37.7 |
| Indirect object identification | After projection | 33.8 |
*Relational facts uses 26 examples, with the same examples for selection and evaluation. Treat this result as exploratory.
Pythia-1B, with hidden states fixed at step 1,000. The readout interventions use the reconstructed final readout; hidden state projection uses the native step-1,000 readout, so the baselines differ. These tests concern readout margins on fixed hidden states.
Feature selection and evaluation use separate halves of the examples for agreement, numeric comparison, and indirect object identification. The relational facts set contains only 26 examples and uses the same examples for selection and evaluation, so that result is exploratory.
Some reconstructed readout baselines are near or below chance at this particular pairing of early hidden states and final readout. Those rows need to be read relative to their own reconstruction baselines. The results do not establish how the interventions affect freely generated text.
04 · Capability evaluation
A separate classifier, called a probe, tests whether the internal representations contain information about whether a subject is singular or plural. We compare its accuracy with the model’s own readout choosing between the corresponding verb forms. “Native readout” means the model’s readout from the same training checkpoint.
The probe recovered this information before the readout reliably chose the correct verb. The gap closed for simple sentences, but persisted at the end of training when additional phrases or clauses separated the subject from its verb. Recovering information with a probe does not, by itself, show that the model uses it.
Dashed horizontal line: 50% chance accuracy. The swept readout is the best alternative checkpoint readout in the sweep.
| Task | Probe % | Readout % | Gap in points |
|---|---|---|---|
| Simple agreement | 100.00 | 100.00 | 0.00 |
| Across a prepositional phrase | 100.00 | 97.15 | 2.85 |
| Across an object relative clause | 100.00 | 90.50 | 9.50 |
| Across a subject relative clause | 100.00 | 75.42 | 24.58 |
Pythia-6.9B. Availability is measured by a linear probe; expression is measured by the readout’s preference between the answer tokens. Probe decodability alone does not establish that the model uses that information in its behavior.
05 · Controlled training
Does the pace of readout learning affect when its features change? The learning rate controls the size of weight updates. In these control runs, each fourfold increase in the readout learning rate halved the training step at which the measured reorganization occurred: a period of change in the feature population.
Shortening warmup, the initial period when the learning rate gradually increases, left the timing unchanged at the baseline rate.
Four runs of a model with 31 million parameters, sharing initialization and data order: three learning rates and a shortened warmup control. One run and one crosscoder fit per condition. Final validation losses agreed within 0.06 nats.
| Multiplier | Reorganization step | Median peak step |
|---|---|---|
| 0.25× | 2048 | 128 |
| 1× | 1024 | 64 |
| 4× | 512 | 32 |
| 1×, short warmup | 1024 | 64 |
