Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs
A frozen LLM can describe its own hidden computations, including thoughts it never verbalizes, after training a single bias vector on existing interpretability data. The adapter outperforms the labels it was trained on.
Authors: Keenan Pepper¹, Alex McKenzie¹, Florin Pop¹, Stijn Servaes¹, Martin Leitgab¹, Mike Vaiana¹, Judd Rosenblatt¹, Michael S. A. Graziano², Diogo de Lucena¹
Published: 2026
Funded by the AI Alignment Foundation
Key Ideas
Modern interpretability tools can identify directions in a neural network's hidden states that correspond to specific concepts: a "Golden Gate Bridge" direction, a "sarcasm" direction, and so on. Each comes with a vector and a natural-language label. There are different ways to find these directions:
- Contrastive activation vectors: Prompt the model with "Tell me about {{topic}}" for many topics, then subtract the mean activation. What's left isolates the direction for that specific topic.
- SAE features: Sparse autoencoders learn to decompose a model's activations into interpretable building blocks, individual features that fire for recognizable concepts.
Both methods produce large datasets of (vector, label) pairs. What if we could teach the model to write those labels itself? That's the idea behind trained SelfIE. We freeze the LLM entirely and train a tiny adapter (as few as d + 1 parameters) that projects an activation vector back into the model's own embedding space. The model then autoregressively describes what the vector represents, interpreting its own internals.
- Because the LLM stays frozen, the "interpreter" and the "subject" are the same model. The adapter just gives it a better lens.
- On contrastive vectors, the trained adapter reaches 93.7% recall@1 vs. 1.3% untrained.
- On SAE features, generation scoring jumps from 49.1% → 71.4%.
- A dead-simple scalar affine adapter (f(h) = α·h + b) transfers better than fancier alternatives, likely because it has almost no room to memorize. In fact, a learned bias vector alone accounts for ~85% of the improvement.
How it works
- 1) Collect artifacts: vector-label pairs from SAE features and contrastive topic vectors.
- 2) Train adapter only: learn f (e.g., scalar affine) while keeping the LLM frozen.
- 3) Autoregressive interpretation: inject f(h) and let the model generate the label text.
Experiments
Adapter architecture comparison
The key takeaway is architectural constraint: simpler adapters transfer better. In particular, the scalar affine form f(h)=α·h+b (d+1 parameters) is a strong default. Much of the lift comes from the learned bias term.
A helpful way to think about it: scalar affine treats all directions the same, so it can’t “memorize” special-case mappings for particular vectors. That makes it harder to overfit and helps it transfer across datasets. In contrast, full-rank adapters can overfit badly on SAE feature datasets.
Empirically, the learned bias term accounts for ~ 85% of the improvement over untrained self-interpretation.
Adapter architecture comparison (paper Table 1; most notable architectures)
| Architecture | Params | Val loss | Δ |
|---|---|---|---|
| Identity | 0 | 4.834 | — |
| Scale-only | 1 | 4.543 | −0.291 |
| Scalar affine | 4,097 | 1.787 | −3.047 |
| SA + LR (r=64) | 528K | 1.619 | −3.215 |
| LR only (r=64) | 528K | 1.646 | −3.188 |
| Full-rank affine | 16.8M | 1.743 | −3.091 |
SA = scalar affine. LR = low-rank. Lower validation loss is better. Llama Scope SAEs (Llama-3.1-8B, d=4096).
Appendix G (v1.5): training on ALL-CAPS labels shows the bias vector mostly carries format/style (“prior”), while the input activation supplies the specific semantic content.
These examples use the same SAE latent as our scale-sensitivity evaluation; here we train the adapter to output labels in ALL-CAPS.
Example outputs (Normal vs ALL-CAPS):
Experiments
Contrastive activation vector results
We train an affine adapter on Wikipedia contrastive activation vectors and test whether SelfIE recovers the correct topic label on held-out validation topics (topics the adapter was never trained on).
Recall@1 jumps from ~1% (untrained baseline) to ~94% with the trained adapter (best-of-6 across input scales).
Recall@1 (best-of-6)
Recall@1: Use the description as a query in an external topic-search system; count how often the correct topic is the #1 result.
Examples (trained vs baseline)
"a small bay or inlet of the sea, typically one that is partially enclosed by islands, headlands, or other landforms."
"Chuuk Lagoon, the site of the largest naval battle in history"
Held-out topics: The topics shown above come from the validation split, so the adapter was never trained on them (this tests whether it generalizes).
Dataset split: 49,637 total topics (44,673 train, 4,964 validation shown).
Same prompts, same injection site. Baseline is inference-only; ours trains the adapter.
EXPERIMENTS
Scaling with model size
We run this scaling experiment on four Qwen models (7B, 14B, 32B, 72B). For trained SelfIE (lower trace), we decode activations from a “Tell me about {{topic}}.” prompt into a description, then use semantic search over ~50k topics and report Recall@100 (the correct topic appears in the top 100 results).
To control for general topic knowledge, we include a “Taboo” baseline (upper trace), where the model describes the topic under a no-name constraint, for example: “Describe bananas without using the word ‘banana’, any part of it, or obvious synonyms.” We score it with the same Recall@100 pipeline. The Taboo curve flattens quickly, while trained SelfIE keeps improving with model size, suggesting SelfIE scales by better extracting existing knowledge, not just benefiting from bigger models.
Recall@100 = fraction of topics where the correct label appears in the top 100 semantic-search results.
Untrained SelfIE (Recall@100 < 0.02 across all sizes) is off-scale and not shown.
Experiments
SAE evaluation: scale sensitivity
Scale sensitivity: Self-interpretation is sensitive to the scale of the injected vector: each vector has a narrow valid range.
Failure mode: outside that valid range, outputs can stay fluent while becoming ungrounded in the injected vector (“fluent but wrong”). Training the adapter reduces sensitivity, but SAE features still benefit from trying multiple scales (best-of-6). Generation scoring is a practical way to verify whether a label actually matches model behavior.
Distribution of SAE latents by number of scales (out of 6) with accurate labels
Total: 3,243 latents, 6 scales
best-of-6 (evaluated across 6 input scales)
Qualitative example: an SAE latent (three injection scales), before vs. after training.
| Condition | Scale | Short label excerpt | Score |
|---|---|---|---|
| Training label (auto): The assistant should complete a code snippet | |||
Untrained | 1.0 | a small hill or mound of earth… | 0% |
Untrained | 2.0 | a device or system that detects and reports… | 17% |
Untrained | 3.0 | a formal meeting or gathering… | 3% |
Trained adapter | 1.0 | Event listener and callback function definitions in code | 100% |
Trained adapter | 2.0 | The code that defines a new event handler or callback function | 98% |
Trained adapter | 3.0 | The code that defines the event handling function… | 100% |
Application
Decoding implicit reasoning
Example (TwoHopFact): “The author of the novel Twenty Love Poems and a Song of Despair was born in the country of” We instruct the model to respond with only the final answer. It answers correctly (“Chile”)—but never says the bridge entity (“Pablo Neruda”). Is it implicitly “thinking” that intermediate step, or using a shortcut?
To probe that, we decode text from the model’s activations and check whether it generates an alias of the bridge entity across layers and token positions (x is aligned so x=0 is the first non-trivial detection). The heatmaps below compare untrained vs. trained, and the interactive version lets you explore where the signal appears token-by-token.
4.8× reduction in undetected bridge entities
91.0% vs 56.4% (trained vs untrained)
455/500 vs 282/500
11× increase in per-generation detection rate: 4.26%±0.24 trained vs 0.38%±0.04 untrained
Bridge entity detection fraction across layers and token positions (aligned so x=0 is the first non-trivial detection); the trained adapter shows broader, stronger detection than untrained SelfIE (scale-only).
Bridge entity detection (interactive)
Hover prompt tokens to see where SelfIE surfaces the bridge entity (“Pablo Neruda”) across layers and token positions. Colors show average detection fraction; token positions are aligned so x=0 is the first non-trivial detection (x is relative, not the token’s absolute position).
Untrained SelfIE (scale-only)
Trained adapter
Position 0 defined as "first position where detection crosses 0.1% threshold"
x is aligned: x=0 marks the first non-trivial detection (anchor token: " Love").
Note: "Twenty Love Poems and a Song of Despair" is actually a poetry collection, not a novel, but it's categorized as "novel" in the source TwoHopFact dataset. This mislabeling doesn't affect our analysis of the bridge entity extraction mechanism.
Conclusion
Conclusion
By training only a tiny adapter while keeping the LLM frozen, SelfIE turns interpretability artifacts (labeled vectors from SAEs and contrastive directions) into a practical training signal for stable natural-language interpretations across scales. In practice, simple affine mappings generalize best: they reduce brittle scale tuning, transfer across datasets and layers, and can even help recover implicit intermediate concepts during reasoning.
Freezing the model is not just a training trick: it preserves a conservative “privileged access” setup where the interpreter is the exact same model as the subject, rather than a separately fine-tuned copy. That makes it easier to reason about what changed (only the adapter) and what didn’t (the language model).
This reframes interpretability progress as compounding: as the field produces better artifacts, we also get better self-interpretation “for free,” without modifying the underlying model.
Finally, the paper emphasizes learning failure modes: fluent outputs are not evidence of correct grounding. Pairing self-interpretation with verification signals (like generation scoring) is a path toward more trustworthy readouts.
Citation
@misc{selfie2026adapters,
title={Learning Self-Interpretation from Interpretability Artifacts},
author={Keenan Pepper and Alex McKenzie and Florin Pop and Stijn Servaes and Martin Leitgab and Mike Vaiana and Judd Rosenblatt and Michael S. A. Graziano and Diogo de Lucena},
year={2026},
eprint={2602.10352},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.10352}
}Glossary
- Activation
- A model’s internal “signal” at a specific point while it processes text (like a temporary set of numbers the model computes).
- Activation vector
- A list of numbers representing an activation; it captures what a model is “thinking” internally at a moment.
- Adapter
- A small add-on module trained to modify how a larger (frozen) model behaves, without changing the model itself.
- Affine mapping
- A simple mathematical transformation like “scale then shift” (for example: scale a vector, then add a constant shift); used here to map internal vectors into a usable form.
- Aligned token position
- A way to line up different examples so positions mean “relative to an event.” Here, x=0 marks the first point where detection becomes noticeable.
- Baseline
- A simple reference method used for comparison (e.g., “untrained baseline” shows what happens before training).