CharlieKey takeaways from your favorite podcasts
← 4 November 2026Latent Space

⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev

Zach argues a software factory should be a concrete, code-defined, measurable agentic system with its own infrastructure stack, replacing vibes-based coding agent use with observe, measure, tune engineering for teams of scale.

Watch on YouTube ↗

Charlie's summary

  1. A software factory is a concrete thing defined in code, with one factory per product surface, not a loose pile of automations

    Zach says many teams call their heavy agent use a software factory without defining it. He wants it concrete: a YAML file listing repos, skills, MCPs, integrations and a workflow like triage, spec, implementation. Engineers feed work in through Slack, Linear, Jira, GitHub, API or MCP. Because it runs as one system, it can be tuned, measured and benchmarked.

    ▶ 3:06
  2. Centralize agent traces so you can measure coding agents with metrics and LLM-judge scorers instead of vibes

    When agents run through one system, Zach says you get a full trace of every conversation and artifact. That enables Dora-style metrics, human-touch counts per merged PR, and cost sliced by person or SDLC stage. Online evals (scorers) use a judge model to grade past runs, for example for redundant tests that bloat the codebase and waste tokens. Teams can write custom scorers for their own failure modes.

    ▶ 6:41
  3. Self-improvement loops: agents read the scorers and open pull requests against the factory definition itself

    Because the factory is code, coding agents can propose changes to it. Zach's example: improvement agents found 54 triage failures where spawning a sub-agent would have helped, and suggested a PR to the factory. The next step he wants is replaying past runs to confirm a suggested change would actually have improved them.

    ▶ 14:23
  4. Benchmark your own real runs across models and harnesses, then feed the results into routing

    Zach says you can replay curated agent runs against a matrix of models and harnesses such as Warp's, Claude Code, Codex or Pi. The result looks like SWE or Terminal Bench results but on your own work, plotted as cost versus correctness. In his example, one model was 95% correct at about six dollars and another 93% at two dollars. Those numbers can drive model or harness routing.

    ▶ 15:54

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.