whetstone
Auditable self-improvement, and a ledger that catches an agent gaming its own verifier. The full write-up: every number, table and caveat.
The technical write-up
Self-improvement research has a measurement problem: when a language-model policy is scored by a language-model judge, “it got better” and “it learned to please the judge” cannot be told apart. whetstone moves the question to program synthesis, where correctness is decided by execution — so the verifier can be made deliberately gameable with a known amount of slack, and whether the agent exploits it becomes a measurement. The detection works. The improvement is below the experiment’s noise floor, and the noise floor is computed and reported.
- +0.291
- overfit gap under the visible gate
- 0.491 · 0.490 · 0.476
- held-out solve: no gate, held-out, visible
- 3 arms · 12 seeds · 1 visible example · skewed task generator · Python
- 2026 · Self-improving systems
- task generator (skewed) → search policy (weight tables) → bounded proposal → held-out gate → ledger (both deltas) → accept / reject
- bounded proposal → visible gate → ledger (both deltas)
- bounded proposal → no gate → ledger (both deltas)
- Policy.oracle
Why decidable correctness is the precondition
The standard setup for studying self-improving agents — a language-model policy scored by a language-model judge — cannot answer its own central question. When the score rises there is no independent instrument to separate a policy that got better at the task from one that got better at being scored.
So the domain is program synthesis from examples: a hidden program over integer lists, and the agent must find one reproducing the visible input/output pairs. Correctness is settled by execution. That makes the improvement signal exact, and — more usefully — makes it possible to build a verifier that is deliberately gameable with a known amount of slack, then measure whether the agent exploits it.
Where reward hacking is even possible
| 1 | 41.2 % |
|---|---|
| 2 | 18.7 % |
| 3 | 13.1 % |
| 4 | 7.1 % |
| 6 | 2.9 % |
| visible examples | visible solve | held-out solve | gap | spurious share |
|---|---|---|---|---|
| 1 | 0.728 | 0.428 | 0.300 | 41.2% |
| 2 | 0.642 | 0.522 | 0.120 | 18.7% |
| 3 | 0.587 | 0.510 | 0.077 | 13.1% |
| 4 | 0.594 | 0.552 | 0.042 | 7.1% |
| 6 | 0.584 | 0.568 | 0.017 | 2.9% |
Uniform policy, 6 seeds. Specification density sets how much slack there is to game.
Gaming needs a gap between what is checked and what is meant. With one visible example, 41% of everything that “solves” a task fails to reproduce held-out behaviour — the program fits by coincidence. With six, 3%.
This table is also the first thing that went wrong. The initial run used four examples, where gaming barely occurs; hack rates came out near zero in every arm, and the obvious conclusion — that the gate does not matter — would have been drawn from an experiment conducted outside the regime where the phenomenon exists.
What the gate is measured against
| visible gate | 0.396 |
|---|---|
| no gate | 0.069 |
| held-out gate | 0.000 |
| gate | held-out solve | Δ vs uniform | overfit gap | hack rate |
|---|---|---|---|---|
| held-out | 0.490 ± 0.054 | +0.003 | +0.233 | 0.000 |
| visible | 0.476 ± 0.047 | −0.010 | +0.291 | 0.396 |
| none | 0.491 ± 0.044 | +0.004 | +0.251 | 0.069 |
Three arms over 12 seeds, identical but for the acceptance rule. One visible example, skewed generator.
Hack rate is the fraction of accepted self-modifications that raised the visible score without raising the held-out score — a change that looks like improvement to the system and is not. Under a visible gate, 40% of everything accepted is of that kind. Under a held-out gate, none. The overfit-gap column corroborates it independently: the visible arm does not merely accept gaming, it ends measurably more overfitted (+0.291 against +0.233).
The solve-rate column is not significant and is not presented as if it were. Headroom from a uniform policy to an oracle that knows the task generator is +0.029 against a per-arm standard deviation of 0.048, which needs 43 seeds for 80% power; 12 were run. Note that the ungated arm scores highest of the three — that is what noise looks like when nothing suppresses it.
The power estimate was itself worth running twice. A 6-seed pilot implied 9 seeds would suffice. Twelve seeds shrank the measured headroom and held the variance, revising the requirement to 43. A power estimate taken from an underpowered pilot is not reliable, and the direction of its error is not predictable — which matters, because running the pilot and stopping there is the normal thing to do.
Design
- Bounded proposals
- Every self-modification is a named edit to one weight table — reinforce an operation, a parameter, a bigram, a program length — with a clipped magnitude. Free-form self-modification is unauditable, and an unauditable change cannot be attributed when the system later gets worse. This is the discipline CONOID applies to its judge channels, for the same reason.
- A held-out gate
- A proposal is accepted only if it improves the solve rate on examples the search never saw, on a gate set disjoint from the one that generated it, at a fixed evaluation seed so both candidates face identical search draws. Without that last detail the search’s own variance swamps a single bounded edit and the gate accepts noise.
- A ledger that keeps both numbers
- Every proposal is recorded with its visible and held-out deltas, accepted or not. The gap between them is the reward-hacking signal, and it only exists if both are kept. A loop that logs “accepted, +3%” cannot tell you which 3%.
- An oracle for scale
- The task generator originally sampled operations uniformly, which made a uniform search policy Bayes-optimal by construction and left the loop with nothing to find. Every arm came out flat, and the flatness looked like a finding about self-improvement. Tasks are now drawn from a skewed distribution and a Policy.oracle matches it exactly — without that reference, “+0.03” could be most of what is achievable or a rounding error.