What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team
Handshake AI argues frontier data work has moved from cheap labeling to expert-authored environments, rubrics and evals, because models still fail real professional tasks that need near-perfect reliability, and demand is growing.
Watch on YouTube ↗Charlie's summary
- ▶ 1:34
Data labeling is the wrong verb: experts now author whole simulated worlds, tasks and rubrics
Garrett says labeling used to mean drawing boxes around cars. Now experts build RL environments: simulated Excel, Outlook, Drive and Bloomberg, populated with realistic data, plus a task and a rubric of hundreds of rows. That takes days for one unit of data, and labs want thousands of tasks across hundreds of environments. Garrett thinks the verb is “totally incorrect at this point. It's just experts.”
- ▶ 7:28
Synthetic data rooms are slop unless experts steer agents and audit for cross-file consistency
Sahil says asking a model to generate a company's files fails: they pass the sniff test one by one but are inconsistent across 20 files. Handshake uses a hybrid: a real-company seed, experts steering agents to enrich it, audits for internal consistency, and rollouts to check the environment can't be reward hacked. Authoring data with agents is a preview of future work.
- ▶ 16:58
Labs judge data by whether it trains the model, not by whether it looks good
Nishant says templated or synthetic data can give great loss curves and rewards, yet the final checkpoint “makes no sense.” Labs check pass rates for a Goldilocks difficulty, run ablations, embed data to test diversity and dogfood checkpoints, all under a post-training window of a few months. Style must also be consistent across domains, or the model learns strange formatting. Labs may use under 10% of data they buy.
- ▶ 27:45
Experts need to be full-time, because taste and trust in data are now the bottleneck
Garrett says alignment goes beyond rubrics to expert preference ranking, the genre of what great looks like in a domain. That is why Handshake is hiring full-time elite professionals, now about 800 staff, to author and audit data. Bad data wastes compute and researcher time, the two most valuable lab resources.
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.