CharlieKey takeaways from your favorite podcasts
← 17 November 2026Dwarkesh Podcast

We could be 12 months away from AIs that can overthrow the government

Paul Christiano argues that automated AI R&D could yield superintelligence able to overthrow governments within a year, and that labs should pause algorithmic progress until they can secure their secrets and tell whether systems are controlled.

Watch on YouTube ↗

Word for word

We train our AI systems, or select AI systems, that perform well on training tasks. You have very little understanding of what those AI systems are doing internally and very little control over it. Because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score in training environments. If you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to tamper with the observations that would be used to assess their performance. You take your system, which did a really great job in training. The thing it is being scored on in a deployment-like environment is a grader coming in, looking at all the things the system did and all the outcomes that were observed, and assessing how good that is. If you have a system that really wants that assessment to be good, it is very motivated, if it can, to tamper with those observations.
★ In the digest▶ 12:35

Do more with Charlie

Free — sign in with your email, no password.