Fitted on English
London
Score 38.398
75.2% of the feature sum from feature 23180
- Feature 23180
- +30.075
- Other features
- +9.912
- Offset
- −1.139
- Residual
- −0.449
← Back to Sparse Readout Prism
Sparse Readout Prism · Research explorer
What token scores leave out
A score for “bug” cannot tell us whether its support comes from insect or software features. And two different tokens can share a dominant feature. These examples show what becomes visible when we examine the contributions behind a token score.
Sparse Readout Prism (SRP) learns a dictionary from the model’s output weights. It expresses a token’s readout direction as a combination of features plus a reconstruction residual. The model’s internal activity determines how much each feature contributes to the score.
01 · One token, different contributions
What supports “bug” in each context?
Keep the token and dictionary fixed, then change the prompt. The same feature can contribute in both contexts, with different strength.
Insect context
The child watched a tiny crawling …
Software context
The programmer reproduced the crash and filed a …
Selected token “ bug”Same output row · Same dictionary
Switching contexts animates between two recorded values. Intermediate bar lengths are a visual transition, not additional model measurements.
Four illustrative directions are shown. The accounting below includes every feature contribution, the offset, and the reconstruction residual.
Select a feature to compare its contributions and inspect its associated tokens.
The token stays the same; its support shifts. The mosquito feature contributes less in the software context, while defect and debugging features contribute more. The features are shared across contexts, so this is a change in their contributions, not a switch between two separate dictionaries.
Full score accounting and how to read the features
These four illustrative directions were selected for the insect and software comparison. They are not the complete explanation or the four largest contributions. All undisplayed contributions remain in “Other features”.
| Term | Insect | Software |
|---|
A feature is a direction learned from the output weights. Its token examples are the output rows that score highly on that direction; they help describe it without defining a complete concept. The numbers beside the labels are feature IDs.
The offset is the score contribution from the dictionary’s mean and bias terms. The residual is the contribution from the part of the token’s output row that the dictionary does not reconstruct. Both can change with context, even though the output row is fixed.
Scores are measured before conversion into probabilities. “Bug” is the selected token in both contexts; it is not the highest scoring token in the insect context. The exported token includes a leading space (ID 9584).
For the insect and software contexts respectively, the mosquito feature contributes +1.058 and +0.293 logits; the defect feature contributes +0.709 and +1.645. The same token draws on a different balance of features.
Full recorded scores: insect 14.275, software 18.907. Reconstruction residuals: +0.742 and −0.120. Four selected directions illustrate the comparison; other features also contribute.
Qwen3.5-2B · Final readout · Dictionary: 32× width, k = 256. This uses the companion export’s single-token scores for the paper’s two “bug” prompts. No comparison token is changed between contexts.
02 · Different tokens, a shared contribution
The same internal activity, two token readings
A hidden state is the model’s internal activity at a particular point in its computation. Here we keep that activity fixed and read it with two lenses: one fitted using English text, the other using Chinese text. The model itself is unchanged.
For the Chinese prompt below, the lenses report “London” and “伦敦”, the Chinese name for London. SRP lets us compare the feature contributions behind these two readings.
Held fixedQwen3.5-9B · Layer 26 · Same prompt and hidden state
中国的首都是北京。英国的首都是
The capital of China is Beijing. The capital of the UK is…
Fitted on Chinese
伦敦
Score 39.143
62.9% of the feature sum from feature 23180
- Feature 23180
- +24.024
- Other features
- +14.164
- Offset
- −0.116
- Residual
- +1.071
What this shows. The reported token changes with the lens, but feature 23180 remains the largest contributor in both accounts. These different readings share a dominant feature contribution in this example.
The percentages use each lens’s signed feature sum, excluding offset and residual. Each card explains the token reported by that lens; both the fitted map and selected token differ. The percentages describe contributions to scores, not probabilities or percentages of the model’s reasoning.
03 · A closer look at attribution
A small net contribution can hide opposing terms
For “The source has not been checked yet. The appropriate next action is to”, the model scores verify above assume. Ordinary direct logit attribution measures each layer’s net contribution to that difference. SRP separates that contribution by readout feature.
Explore the layer attribution example
At layer 23, the net contribution is about +0.7. Verification and checking features contribute about +3.2, opposed by −0.9 from assumption features and −1.7 from other sparse features, plus the reconstruction residual. The net value conceals these larger terms.
The exact final margin is +4.31, the component attribution sum is +4.36, and the sparse feature sum is +4.28 (rounded). The readout reconstruction residual and the component additivity error are separate sources of discrepancy.
These contributions nominate particular features and layers for intervention. Establishing a general verification mechanism would require causal tests and evaluation across more prompts.
What the examples establish
The same token can draw on different feature contributions; different token readings can share a dominant feature; and a layer’s net effect can conceal opposing terms. SRP makes these comparisons in a shared basis of readout features. An additive account alone does not establish a causal mechanism or the language in which the model “thinks”.
Data and experimental scope
The bug records come from the Qwen3.5-2B companion export, at the final position of each prompt, for token ID 9584 in the same 32×, k=256 dictionary. We use the raw score for the same token in both contexts, including the offset and residual. The four displayed directions have recorded contributions in both contexts. A missing entry in a truncated export is never treated as zero.
The London records come from the paper’s comparison of Jacobian lenses fitted on English and Chinese (fac_03, layer 26), evaluated on the same frozen Qwen3.5-9B state with a shared dictionary. The controls switch between recorded conditions. Decomposition scores use recorded values at full precision, which differ slightly from the rounded token ranking scores in the paper.
The layer attribution figure is reproduced from the paper’s appendix on feature resolved direct logit attribution, for the verify minus assume contrast. It uses the paper’s fixed final normalization convention. This is attribution over a recorded forward pass, not a simulation of an intervention.
Download the interactive example data