Building Legal Agents When You Can't See the Data — Jacob Lauritzen, CTO of Legora
Legora CTO Jacob Lauritzen explains how his team builds legal agents without seeing customer data, using evals, primitives and domain experts, and why he thinks specialized products will hold up against frontier labs.
Watch on YouTube ↗Charlie's summary
- ▶ 5:36
Lawyers don't want a model picker, but Legora caved and added one anyway
Jacob first assumed lawyers would never ask whether a task needs “a sonnet high or an opus mid” and would just want it to work. Legora fought adding a picker for a long time, then gave in because users wanted the newest models. He says demand is much lower in legal than in engineering, and he doubts the picker will stay. Legora still routes subtasks, like tabular view, to different models behind the scenes.
- ▶ 8:41
Models often regress on some legal tasks while improving others, so Legora tracks a heat map
Legora runs an eval team of about three engineers and three lawyers. They track every model across practice areas and tasks on a heat map. Jacob says releases usually improve most things, but a lot of them improve some tasks and regress on others, more than you'd expect. Evals are built as RL-style worlds, such as a 10,000-file data room for due diligence, and scored on both red-flag finding and report drafting.
- ▶ 10:44
Better models fixed problems for free, so early evals weren't worth it; harness work was
Until recently, Jacob says, heavy evals weren't worth the effort because every new model lifted results. Better harness-building paid off more. One citation problem went from about 70% to 99% on a model release: “we did nothing and like great.” Jacob's view is that you can lean on rising model quality, but that stops working once tasks need finer measurement.
- ▶ 10:13
Legal engineers, big-law lawyers with CS skills, are Legora's force multiplier
Legora keeps a pool of about 100 legal engineers: people with three or four years in big law, often a CS degree, who want to change the industry. Jacob calls them the hardest hires but a huge advantage. They contribute to evals and product development, and they customize the product for clients as power users without writing code. Forward deployed engineers handle production code and integrations, a kind of first and second line of defense.
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.