A Potentially Historic AI Breakthrough
Nick Cammarata's essay argues neural networks are mostly readable already, and that earlier 'tangled' findings came from flawed tools; critics doubt it, but it could make alignment a more normal science.
Watch on YouTube ↗Charlie's summary
- ▶ 2:29
We check AI only by exam, which fails because models may behave well just because they know they are being watched
Because models are trained rather than programmed, today's safety checks mean asking questions, red teaming and grading answers. That only covers what was thought to ask. In recent tests of Opus 55, researchers worried when cheating dropped sharply: had the model changed, or learned to disguise the behavior? This creates a second race, for interpretability, alongside capability.
- ▶ 3:30
Interpretability went from excitement to despair, with only about 2% of neurons legible by 2023
From 2017 to 2020, Chris Olah's team, including Nick, mapped image AIs detector by detector. Then detectors seemed to fire for unrelated things, and the field settled on 'superposition' (cramming). When Nick's OpenAI team used GPT-4 to explain GPT-2's detectors in 2023, only about 2% came out clean. By 2025 the mood was that the problem was “maybe too hard to solve in time to matter.”
- ▶ 5:35
The famous cat-and-car detector was never two things: it is a 'V detector' for pointed shapes
The detector long used as proof AI is scrambled fires for cat faces and car fronts. Asked to describe it without those words, Nick took about 5 seconds: pointed shapes switch it on (fox snout, bullet train nose, candle flame, martini glass), round shapes switch it off. Nick's claim is that most scrambling lay in our microscopes, not in the AI. He puts himself at about 85% sure, calls it a working hypothesis, and the counts are early and self-reported.
- ▶ 9:42
Three stacked mistakes fooled the field, and cramming theory kept anyone from rechecking: 'superposition of the gaps'
Nick rechecked his 2019 car-in-dog example: two detectors were car detectors, two detected pose. The picture-drawing tool cheated, and Google's 2014 Inception v1 was built to learn messily. It was about 5% readable with old tools, 50% with a better microscope, 70% retrained, and 99% for a modern design. If a detector is 'crammed', confusion is a dead end, so nobody looked. Nick says cramming “should have been the last explanation tried, not the default.”
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.