CharlieKey takeaways from your favorite podcasts
← 8 October 2026The AI Daily Brief

The Best Way To Test New AI Models

Published benchmarks tell you little about a model's fit for your work, so build a repeatable, blind, personal benchmark of your own tasks and use it to decide whether to switch, split or stay.

Watch on YouTube ↗

Charlie's summary

  1. Published benchmarks and early hot takes are weak guides; the only real test is using the model on your own work

    Nathaniel is skeptical of published benchmarks because many are now in training data, and the longer they exist the less they're worth. Even perfect ones miss how a model feels and fits your stack, especially for subjective work like writing. Nufar adds that early buzz skews toward toy games and 3D demos. Real reports, like a model quietly dropping a condition in a contract review, arrive about a week in. Test it yourself.

    ▶ 1:32
  2. Run a five-step personal benchmark: pick tasks, pick candidates, run, blind-score, then decide

    Nufar's process starts with the most important step, choosing which tasks go into a repeatable benchmark. Next choose candidates, which can be models, harnesses or agentic tools. Then run them manually, with a script, or on an eval platform. Hide which model made each result. Finally decide, because winning a run doesn't automatically mean you should change. Once built, you rerun it whenever a significant shift comes along.

    ▶ 4:13
  3. Personal benchmarking is worth it even if the frontier is good enough or your company limits your tools

    Nufar answers two objections. If everything is already good, a new model may still unlock a use case that was only on your wish list, so check periodically rather than weekly. If your company restricts you to certain tools, there is still choice within them, such as fast versus thinking modes. A benchmark on mock data can also make the case for buying another license.

    ▶ 8:27
  4. Choose four to six tasks that are diverse, frequent and judgeable, plus one or two wish-list tasks

    Nufar says to mirror your real balance of work and personal use and pick tasks you trust enough to act on. Favor regular tasks, and ones where you can spot bad output quickly, since that is often easier than defining good. Include one or two wish-list tasks AI has failed at before. Her own list includes LinkedIn posts, a pricing pushback email, and a weekly plan that no model has yet gotten right in one shot.

    ▶ 11:05

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.