SelfIE across tasks: reading a model’s hidden concepts

SelfIE adapters translate a model’s internal activations into language, which could help check whether its explanations reflect its computation. After training an earlier Qwen adapter on Wikipedia topic vectors, Keenan used the newly released WorkspaceBench to test a broader question in a quick research sprint: could one adapter learn to read several kinds of unstated concepts?

In one WorkspaceBench task, a prompt describes a concept, such as “prison,” without naming it. The reader must recover the implied concept from the model’s activations.

The underlying model was Qwen3.6-27B, with its weights frozen. Only the 717,000-parameter adapter was trained. The same adapter read activations across layers 20 to 60 and seven task types: associations, two-hop facts, relationship chains, intermediate arithmetic results, rhyme plans, typo corrections, and instructions to think about or avoid a concept.

Training used five folds: each run trained on four parts of the data and evaluated on the remaining part, so every evaluated item was held out from that run’s training. This tested new questions from the same task families, rather than entirely new families.

Against J-Lens, SelfIE achieved a 76% average pass rate versus 33%, a gain of 43 percentage points (95% confidence interval: 39 to 48 points). An item passed if any read layer named its hidden target. SelfIE scored higher on six of seven tasks; the two-hop fact comparison showed no detectable difference.

That comparison used 640 matched items, with the same read layers, positions, and benchmark judge for both methods. Six task families used five folds; the think/avoid comparison used only 20 items from one fold.

The Oracle Lens comparison was a separate experiment: both methods trained on the same items, batches, and labels for three epochs. It used two folds, with 40–48 items per task, except think/avoid, which again used 20 items from one fold. Oracle Lens trained a 117-million-parameter LoRA, about 163 times the size of the SelfIE adapter.

Five of the seven task comparisons showed no detectable difference. Oracle Lens did better on arithmetic and think/avoid instructions. Across the six tasks excluding think/avoid, its average advantage was two percentage points (95% confidence interval: −2 to +6), with both methods at 100% on two tasks. This does not establish that the methods are equivalent.

Keenan highlighted a scoring ambiguity in prompts that ask the model to focus on a concept or avoid it. In one example, Qwen copied the sentence “The account settings were saved successfully.” while receiving either an instruction to concentrate on envy or an instruction to avoid thinking about it.

SelfIE named envy, envious, or jealous in 15 of 77 layer/token readouts for the concentrate instruction, and in none of the 77 readouts for the avoid instruction. These counts describe the illustrated example, not an aggregate success rate across prompts.

The benchmark rewards naming the target even on avoid prompts, favoring readers that name it more often. Keenan pointed to this scoring rule as a possible explanation for Oracle Lens’s advantage on the task. A readout that omits envy does not, by itself, establish that the model successfully avoided the concept.

The original SelfIE adapter paper and code explain the underlying method. WorkspaceBench provides the task designs and evaluation framework used in this exploration.