Zain Ali

← All work

What didn’t work

A portfolio of only successes isn’t proof of good judgement. Here is everything in these projects that didn’t work, 33 things in all, each with the number that showed it.

gedanken

Does an AI check an idea before it trusts it?

  1. End to end, it never beats chance0 of 70 pairs right in bothresult
  2. Handing it the thought experiment helpsgain −0.021guess that was wrong
  3. An RL agent learns to checkbest median 0.495, bar 0.75guess that was wrong
  4. OpenAI tools sent in strict mode156 of 156 experiments refusedbug
  5. A sandbox that could read other processes’ environments2,133 python calls, none used itbug
  6. Unbilled reservations counted as spend38 → 64 pairsbug

whetstone

When an AI improves itself, can you tell a real gain from gaming its own test?

  1. Improvement below the noise floor43 seeds for 80 % power, 12 runresult
  2. A power estimate from an underpowered pilot6 seeds implied 9estimate that was off

loci

Does where a memory is filed decide what it can recall, and what it can imagine?

  1. Filing for recall, read for imagination0.24–0.31 against 0.95–1.00result
  2. The rule against the oracle0.011–0.057 belowresult
  3. A memory that files itself0.57–0.69idea that didn’t help
  4. The theory’s worst row0.068result
  5. Accuracy is silent about structureguess that was wrong
  6. Consolidation discovers structureguess that was wrong
  7. The price of imagination is recallguess that was wrong
  8. Round 1’s criteria8 of 17result

engram

Memory for AI assistants, inspired by the brain.

  1. Reinstatement and sleep on the public benchmarks0 of 470result
  2. v0.1’s lead over BM25−0.012bug
  3. The schema store with a lexical encoder0.287 → 0.613idea that didn’t help
  4. Lateral inhibition in the dentate gyrus0.1277 → 0.1284idea that didn’t help
  5. Eviction coupled to promotionidea that didn’t help
  6. DG/CA3 as a ranking pathway0.048 top-1idea that didn’t help
  7. The lead on LongMemEval+0.003 [−0.023, +0.029]result

spindle

A faster engine for running language models.

  1. The cache inferred positions16.4 off HuggingFace in max absolute logitbug
  2. A stale distribution in speculationbug
  3. The batching speed-up, under its control1.16×result

reverie

An AI that plans by imagining the future.

  1. Planning barely beats randomresult
  2. Imagined rollouts degrade with horizon0.635 → 0.530 F1result
  3. Sampling imagined futures should beat argmaxF1 0.553 vs 0.604guess that was wrong
  4. The parallel-token shortcut is the cause0.538 vs 0.553, 6× slowerguess that was wrong
  5. The model never saw a terminationrecall 0.000 through 3,000 stepsbug
  6. Unweighted BCE on a 1.6 % positive class98.4 % accuracy, zero recallbug
  7. Cell accuracy as a reconstruction metric0.92 by predicting nothingbug