CharlieKey takeaways from your favorite podcasts
← 2 November 202620VC with Harry Stebbings

Modal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute

Modal CEO Erik argues inference is the new cloud, that it will fragment into specialized products best served by a broad platform, and that demand for compute and software will keep growing despite pullbacks.

Watch on YouTube ↗

Charlie's summary

  1. Inference alone is a commodity; Modal wins by bundling a broad platform of products that are hard to copy

    Erik thinks selling tokens for dollars means no switching costs and a “race to the bottom.” Modal couples inference with sandboxes, post-training, reinforcement learning and batch jobs, built on its own file system, container runtime and scheduler. People want “a portfolio of different things,” not a single-trick pony, and that infrastructure is very hard to replicate.

    ▶ 11:40
  2. Software margins are gone because every layer of the AI stack takes 20-30%

    Erik says a dollar paid at the top flows through applications, models, infrastructure, cloud and Nvidia, and each layer keeps maybe 20-30% gross margin. The 90% software margins of a decade ago are gone. He thinks Modal can reach 40-60% margins long term, but never software-level margins.

    ▶ 13:42
  3. Inference will fragment into many specialized sub-markets, which Modal wants to host under one roof

    Erik says the market became 10x or 100x bigger than anyone expected, which leaves room for low latency, batch, fine-tuning, new hardware and specialized models. He sees a chance to “deconstruct the cloud and rebuild it for this new era.” He likewise says a company like Jeb (specialized models for classification and regression) could be a $100 billion company.

    ▶ 14:42
  4. Pooling many customers' bursty demand into one GPU pool lets Modal sell pay-per-second capacity without planning

    Erik says Modal works with 25 neoclouds and treats the world's compute as one pool. Individual customers are unpredictable, but in aggregate demand is smooth. Customers skip multi-year reservations and pay per GPU second. Modal forecasts by extrapolating 30-40% monthly growth over the 3-4 month lead time. He believes prices will eventually fall, because “hardware prices always go down in the very long run.”

    ▶ 17:15

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.