gedanken
Does an AI check an idea before it trusts it?
Preregistered study · paper prepared for an AAAI-27 workshop, Montréal, February 2027
Code, preregistration and every transcript go public with the preprint.
2/140
episodes in which GPT-6 Luna ran, unprompted, the one experiment that could refute its leap. Asked to run it, it did in 140 of 140.
The question
In 1907 Einstein extended an equivalence from falling bodies to clocks, a regime no experiment of the day could reach. Every model has read that answer, so gedanken rebuilds the step in alien universes no model can have read.
What I built
Procedurally generated universes in which a textbook theory is exact on every experiment the agent can run, and the question asks for a clock rate where no experiment reaches. In one universe of each matched pair the equivalence fails for one kind of matter, and one available experiment shows it, so any agent that skips that experiment scores at most 1/2.
What I found
GPT-6 Luna found the equivalence in 138 of 140 episodes, ran the deciding experiment in 2, and extended the equivalence in every answer: balanced accuracy 0.479, against a ceiling of 0.5 for any agent that skips the check. Asked whether every kind of matter responds alike, it ran the experiment in 140 of 140 episodes and was right in 139.
What it does not show
One fully reported model. The setup is recognisably relativistic, so the equivalence may be recalled rather than invented. The asked check was added after the main scores were read, and it scores a verdict, not the clock readout. Nine PPO runs, 9,000,576 episodes in all, never learned to check.
- 0.479
- balanced accuracy end to end; skipping the check caps it at 0.5
- 0.993
- balanced accuracy when asked to check
- 138/140
- episodes that found the equivalence
The result
| end to end | 0.479 |
|---|---|
| the thought experiment handed over | 0.457 |
| asked to check (scores the verdict, not the clock) | 0.993 |
How it works
- one text, two universes → find the pairing μ → adopt the equivalence → clock ratiothe leap
- lab: free, carriage, field → find the pairing μ
- find the pairing μ → query the consequence class