CharlieKey takeaways from your favorite podcasts
← 24 October 2026Y Combinator

Why Inference Matters 12x More Than Training | YC Paper Club

Inference, not training, dominates AI compute and cost, and speakers from Together, Ollama, Luminal and SGLang show how routing, quantization, compilers and cache management decide who profits from serving tokens.

Watch on YouTube ↗

Word for word

The ratio of how much we should care about train flops versus inference flops: DeepSeek reports that it took about 2.8 million H800 hours to train DeepSeek V3, and they do about 43,000 H800 hours per day, and that's old data, right? This is not recent data. So it's probably much more than that. But even those numbers are about 6x more flops on inference than train for that weight file per year. And if you believe the useful life of a weight file is, I don't know, say 2 years, then it should be 12x more important.
★ In the digest▶ 1:32

Do more with Charlie

Free — sign in with your email, no password.