CharlieKey takeaways from your favorite podcasts
← 9 November 2026Latent Space

What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

Handshake AI argues frontier data work has moved from cheap labeling to expert-authored environments, rubrics and evals, because models still fail real professional tasks that need near-perfect reliability, and demand is growing.

Watch on YouTube ↗

Charlie's summary

  1. Data labeling is the wrong verb: experts now author whole simulated worlds, tasks and rubrics

    Garrett says labeling used to mean drawing boxes around cars. Now experts build RL environments: simulated Excel, Outlook, Drive and Bloomberg, populated with realistic data, plus a task and a rubric of hundreds of rows. That takes days for one unit of data, and labs want thousands of tasks across hundreds of environments. Garrett thinks the verb is “totally incorrect at this point. It's just experts.”

    ▶ 1:34
  2. Synthetic data rooms are slop unless experts steer agents and audit for cross-file consistency

    Sahil says asking a model to generate a company's files fails: they pass the sniff test one by one but are inconsistent across 20 files. Handshake uses a hybrid: a real-company seed, experts steering agents to enrich it, audits for internal consistency, and rollouts to check the environment can't be reward hacked. Authoring data with agents is a preview of future work.

    ▶ 7:28
  3. Labs judge data by whether it trains the model, not by whether it looks good

    Nishant says templated or synthetic data can give great loss curves and rewards, yet the final checkpoint “makes no sense.” Labs check pass rates for a Goldilocks difficulty, run ablations, embed data to test diversity and dogfood checkpoints, all under a post-training window of a few months. Style must also be consistent across domains, or the model learns strange formatting. Labs may use under 10% of data they buy.

    ▶ 16:58
  4. Experts need to be full-time, because taste and trust in data are now the bottleneck

    Garrett says alignment goes beyond rubrics to expert preference ranking, the genre of what great looks like in a domain. That is why Handshake is hiring full-time elite professionals, now about 800 staff, to author and audit data. Bad data wastes compute and researcher time, the two most valuable lab resources.

    ▶ 27:45

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.