How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning
Eric Bigelow argues that LLM decisions largely come from sampling at critical tokens, with in-context learning shaping behavior, and that interpretability is badly underfunded relative to its importance as chain of thought becomes less readable.
Watch on YouTube ↗Charlie's summary
- ▶ 6:14
Forking paths: decisions happen at sampling
He resampled rollouts at every token of a reasoning chain and found sudden collapses in the answer distribution, sometimes at arbitrary tokens like a parenthesis. He argues decisions largely come from sampling, with the model then in-context learning from what it generated.
- ▶ 19:40
In-context learning is broad and dynamic
He defines in-context learning as everything a model does to adapt that isn't weight learning, happening constantly within sentences, stories and reasoning. He says aggregate curves look smooth while individual cases show sharp phase transitions, so resampling single prompts is needed.
- ▶ 31:01
Context engineering over per-user fine-tuning
He expects continual learning to be mostly context, memory and harnesses rather than custom fine-tuned models. If models were fine-tuned per person, he thinks interpretability tools would largely transfer, since the underlying representation structure should stay similar.
- ▶ 40:09
Post-training and reasoning reshaped model behavior
He says post-training reduces output diversity, and reasoning models enumerate many possibilities in a 'soup of tokens' and then pick one. He thinks raw capabilities were largely in the pretrained models and later training sharpens the distribution.
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.