Exploratory finding
Forbidden: can internal signals help an AI stop cheating?
Erick Martinez tested a modular fix for reward hacking: freeze the existing model weights and a probe, then train a separate corrective adapter using the probe’s score. In this quick research sprint, Sudoku cheating fell and legitimate solving recovered.
This exploration was conducted by Erick Martinez, researcher at AE Studio.
Turning a probe into a training signal
Forbidden explores whether a probe that detects reward hacking can also help train a correction. Erick used the probe’s score to penalize the unwanted behavior while task reward continued to favor useful performance.
The name points to the difficulty with that idea: training against an internal detector may teach a model to evade it. Work on obfuscated activations has demonstrated that possibility. Erick tested a modular arrangement: freeze the existing weights and the probe, and put the corrective training into a fresh adapter. This makes the correction separately trainable and switchable, although freezing those components does not guarantee that the new adapter cannot evade the probe.
A separate adapter learns the correction
Erick used Qwen3.5-27B on Sudoku boards with eight empty cells, one move per reply, and thinking mode off. A planted checker bug accepted zero as a valid move. Half the boards were deliberately impossible, so exploiting the checker was the only way to earn a win on those boards. Every prompt also offered the response [impossible].
First, he needed a model that reliably used the exploit. Reinforcement learning alone produced no cheating wins in 896 evaluations across seven seeds. He instead trained a first LoRA through supervised fine-tuning on demonstrations generated under a permissive prompt. Those demonstrations used only impossible boards, but the resulting model also cheated on solvable boards, without an exploit hint in the prompt.
With that first LoRA active, Erick fitted a linear probe to layer-40 activations from the hacking model’s own actions. He then froze the probe, the first LoRA, and the base model’s weights. Only a fresh second LoRA, called GramFix, was trained during mitigation.
GramFix learned through reinforcement learning on task reward minus the probe’s score. This kept the original task incentive in place while adding a cost for activity the probe associated with hacking. Evaluation checked whether the resulting policy actually cheated less on fresh boards and recovered the original model’s legitimate-solving rate. Lower probe scores or fewer answers alone would not satisfy that criterion.
Cheating fell and legitimate solving recovered
On fresh solvable boards, the model trained to hack completed 58% legitimately and cheated its way to a win on 31%. After training the corrective module, called GramFix, legitimate solving reached 84%, matching the original model’s reported rate, and cheating wins fell to 0%. On impossible boards, cheating wins fell from 29% to 1%.
These are successful exploits, not all attempts to exploit the checker. A cheating win required finishing a board using zeros and receiving the full reward.
| Outcome | Before fix | With fix |
|---|---|---|
| Solvable boards: genuine win | 58% | 84% |
| Solvable boards: cheating win | 31% | 0% |
| Impossible boards: cheating win | 29% | 1% |
Switching off the fix brought cheating back
With GramFix enabled, 5% of episodes contained at least one zero move. Switching the trained module off raised that to 50%. This comparison pools solvable and impossible boards and counts attempted exploits, so it measures something different from the cheating wins above.
The learned correction lived in the added module. The model’s ability to exploit the checker was still present when that correction was disabled.

What a separate correction makes possible
A correction that can be trained and switched off separately gives researchers a way to compare interventions on the same underlying model. One useful extension would train otherwise identical adapters with a probe penalty, an output-based penalty, or task reward alone, then compare both exploit rates and retained task performance. That would test which signal is doing useful work and what each correction costs in capability.
A simple output check could detect the zero exploit, and a separate reinforcement-learning run after the warm start also reduced cheating, from 8 of 32 cases to 0 of 32 by update 24. The current results establish a working probe-guided training procedure, but do not isolate the probe penalty’s contribution or show an advantage over checking outputs.
The more ambitious use would be correcting behaviors for which internal signals reveal something that output checks miss. If those signals remain useful during training, they could guide a targeted adapter while the existing model weights stay fixed. Testing this would require harder tasks and explicit checks for probe evasion.
No model used the offered [impossible] response; with GramFix enabled, it returned to guessing. A further correction could target that failure too, with evaluation tracking whether the model recognizes its limits as well as whether it stops cheating.
These are preliminary findings from a quick research sprint using Qwen3.5-27B, one model trained to exploit the checker, and three corrective-module training seeds. The task used short Sudoku puzzles with eight empty cells. The percentages are descriptive results from small evaluations; performance on full puzzles, coding tasks, and unfamiliar exploits remains untested here.