CharlieKey takeaways from your favorite podcasts
← 17 November 2026The Generalist

Baseten’s CEO: Inference Is AI’s Final Market

Tuhin argues inference is AI's final market, far harder than it looks, and that companies will move from closed models to owned, post-trained model portfolios while compute stays scarce.

Watch on YouTube ↗

Charlie's summary

  1. Baseten threw away three years of work in six weeks to focus on the inference engine it had been giving away

    Tuhin and his co-founders first built a general machine learning tools company for data scientists, with small models and internal tools. They gave away inference as a commodity because serving on CPUs was trivial. Then an early customer arrived with a fine-tuned Stable Diffusion model (Refusion) that needed a lot of GPUs and took off on Hacker News. Serving big models on big GPUs turned out to be incredibly difficult. In about six weeks they cut the other scaffolding and focused on serving.

    ▶ 3:08
  2. Inference is the final market: even with AGI, the only thing left to do is run the intelligence

    Tuhin thinks that if AI eats the world, all that matters is how the AI is run. His rough math: about $5 trillion of AI spend in 2030, at roughly 60% gross margin, leaves around $2 trillion of inference cost of goods. He compares that to AWS, which he guesses would be worth $2 to $3 trillion as a spin-off, and sees about 10x that opportunity. He calls the numbers made up, but says the pie is big enough to support many inference companies.

    ▶ 7:15
  3. Hyperscalers haven't won inference because it is a new workload that cuts across four of their services

    Tuhin gives three reasons. It is a fundamentally new workload. Hyperscaler commitment models can take months to negotiate, while customers work in days or hours as new models ship weekly. And inference bridges compute, networking, observability and workflow, which hyperscalers run in separate orgs. He says this bridging is where companies can beat them. Neoclouds that did training also found they can't reuse the same stack, which “needs a fundamentally new stack.”

    ▶ 13:53
  4. The customer journey runs from closed models to owned intelligence through a loop of production data and retraining

    Tuhin describes four stages. Customers start with closed models, add open-weight ones, move to dedicated deployments for specific latency or throughput needs, then post-train custom models on their own data. Models in production generate data that retrains the models, and that loop is continual learning. Proprietary data, workflows and user feedback feeding model improvement is what Tuhin calls owned intelligence, and it is what Baseten sells.

    ▶ 20:31

Give your email to unlock the rest of the key points — for every episode.

Do more with Charlie

Free — sign in with your email, no password.