Inside OpenAI DevDay: Superhuman Computer Use, Decisions API, and the AI Cloud — Ari & Nikunj
OpenAI's computer use agents have become faster than average humans, and the API team is exposing agent capabilities, a Jev-inspired decisions API, and caching and compaction primitives for builders.
Watch on YouTube ↗Charlie's summary
- ▶ 13:40
The next frontier for computer use is superhuman speed, which enables realtime products and makes agents the default
Ari says computer use now beats the average human on most tasks. The next goal is to match or beat expert users. That would allow much more realtime products and lower the activation energy of delegating, so people “default to doing certain things in agents” they now do manually. He notes that as agents speed up, bottlenecks shift to things like waiting for sites like DoorDash to load.
- ▶ 7:21
Computer use got better because models debug themselves and agents write code, not just click screenshots
Ari says the biggest change in a year is that models now debug, retry and introspect instead of failing when they hit problems. Agents also write JavaScript that performs many actions at once. They mix screenshots, accessibility data and Playwright by task. Direct access to the DOM lets the model see a whole page, ending the slow scroll-and-screenshot loop. His meal-prep order dropped from two hours to 15 minutes.
- ▶ 11:33
App shots pass the accessibility tree, not a screenshot, giving the model full context
The double-command shortcut looks like a screenshot but attaches the raw accessibility representation. A screenshot of a web page omits link destinations, and calendar event titles can be truncated. An app shot gives the model all of it. Ari says the team worked to make it token efficient, and that screen-reader technology built for humans also suits LLMs.
- ▶ 17:19
Use OpenAI's own computer use harness through the Agents API, and build trust with consent and limited access
Ari says computer use works with any third-party site, and the models are trained on OpenAI's harness. Building on that in-distribution version may bring speed, cost and accuracy advantages. He urges safety checks, user consent before consequential actions like payments, and restricting the agent to the sites and apps it needs.
Give your email to unlock the rest of the key points — for every episode.
Do more with Charlie
Free — sign in with your email, no password.