CharlieKey takeaways from your favorite podcasts
← 17 November 2026AI & I

Giving Everyone AI Tools Isn’t Enough

Bringing AI into established companies means aiming past faster work, extracting tacit human knowledge, measuring value with evals, and driving adoption through good products and concrete demos rather than mandates.

Watch on YouTube ↗

Word for word

It's much easier if you aren't relying on the models being smarter than you and are instead relying on the models having more patience and time than you. So we often have these layered evals, right? There's a system and we have evals for that system, but some of those evals are going to have AI judges, and for the AI judges we then have evals, and sometimes that's a little bit of a recursive process. And so if you have an agent that's watching these traces, you want to train it, right? So you want to read those transcripts yourself and rely on the fact that you are very smart and you can figure things out. You're essentially spot-checking and finding out discoveries for yourself, and then turning that into evals and climbing on that until the system is performing at your level. But now, instead of just doing however many you could do in 10 or 100 hours, it's looking at thousands and thousands of hours.
★ In the digest▶ 27:36
The traditional wisdom in software engineering is that you should never do a full-scale rewrite of a production system. That has been the traditional wisdom for a very long time because these projects almost always fail often catastrophically and waste a lot of human time. However, this is no longer the case. for instance, like recently, Bun, which is the JavaScript runtime behind quad code, and that is like a big part of Enthropic, they rewrote their whole 1 million line codebase in Rust fully autonomously in about a week. and then like a few more weeks for verification. this is a project that literally would have taken a team over a year
★ In the digest▶ 5:21
Even if the long-term vision is to take people completely out of the loop, there's probably an intermediate period where we start with a system that's running in shadow mode, just trying to track what's going on. And then we get to a system that is pre-doing a lot of work and letting humans approve it, that sort of thing. And in all of these services, we're going to track every single interaction that happens. And when people correct things, we're going to notice that. A great thing is to have really smart models watching the transcripts of everything that happens. And you don't have to work on something that's just a traditional observability platform, where you're looking for signals ahead of time. You can have an agent that is looking at all of the traces and seeing if it can pick out something that you didn't know you were looking for yet.

Give your email to unlock the rest of the quotes — for every episode.

Do more with Charlie

Free — sign in with your email, no password.