CharlieKey takeaways from your favorite podcasts
← 24 October 2026Y Combinator

Why Inference Matters 12x More Than Training | YC Paper Club

Inference, not training, dominates AI compute and cost, and speakers from Together, Ollama, Luminal and SGLang show how routing, quantization, compilers and cache management decide who profits from serving tokens.

Watch on YouTube ↗

Charlie's summary

  1. Care about inference flops about 12x more than training flops when judging models

    Using DeepSeek, the only company publishing its data, Francois estimates about 6x more flops go to inference than training per year for one weight file. If a weight file's useful life is roughly 2 years, that becomes 12x. Papers that plot only train flops (as in Chinchilla) should plot iso-inference flops instead. He says it matters most for COGS if you serve models.

    ▶ 1:32
  2. Inference is hard because decode is memory-bound and GPUs run far below their advertised flops

    Francois notes that training and prefill might reach about 50% model flop utilization, and decode maybe 5%. DeepSeek's reported numbers show prefill at 1.5 million tokens per second per GPU with cache hits versus 14,000 for decode. Their 96% cache hit rate comes from keeping KV caches for 72 hours. Agent traffic caches well; users who start fresh chats each time do not.

    ▶ 3:35
  3. Efficiency gains come from three buckets: model, kernel/runtime, and harness innovations

    Francois groups the ideas. Model: MoE, SSMs, latent attention, quantization, distillation. Kernel/runtime: flash, radix and page attention, many borrowed from OS and database theory. Harness: routing easy requests to smaller models, speculative decoding, recursive language models that don't load full context, and compaction. Each saves flops without needing new hardware.

    ▶ 8:13
  4. A request's life is routing, tokenization, scheduling, then prefill and decode, which are very different workloads

    Dan explains that routing must balance load and also send repeat requests to the GPU already holding their KV cache. Prefill is compute-bound with high arithmetic intensity. Decode must reload the whole model weights for every token, so it is memory-bound. Together therefore thinks about disaggregating them onto different GPUs with different kernels and parallelism.

    ▶ 16:24

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.