It's much easier if you aren't relying on the models being smarter than you and are instead relying on the models having more patience and time than you. So we often have these layered evals, right? There's a system and we have evals for that system, but some of those evals are going to have AI judges, and for the AI judges we then have evals, and sometimes that's a little bit of a recursive process. And so if you have an agent that's watching these traces, you want to train it, right? So you want to read those transcripts yourself and rely on the fact that you are very smart and you can figure things out. You're essentially spot-checking and finding out discoveries for yourself, and then turning that into evals and climbing on that until the system is performing at your level. But now, instead of just doing however many you could do in 10 or 100 hours, it's looking at thousands and thousands of hours.
★ In the digest▶ 27:36