A look inside the scheduling, caching, and routing work that took our median agent response time below the 100ms threshold — and what it unlocks for real-time products.

When we set out to bring agent response times under the 100ms threshold, our median was sitting at 240ms. That number was an aggregate of three distinct problems — and solving it meant treating latency not as a metric to monitor, but as a property to design around.

Where the milliseconds were going

Our first move was to instrument every stage of the request path with the same tracing layer we ship to customers. The breakdown was illuminating: roughly half the time was spent in scheduling and queueing, a third in model inference itself, and the rest in serialization and network hops.

Once we could see the shape of the problem, the work became obvious. You can't optimize what you can't measure, and the trace made it clear that inference was not actually our bottleneck.

Scheduling overhead

Agents in a multi-step workflow were being re-queued between every tool call. Each hop paid the full cost of cold scheduling. By keeping a warm execution context alive for the duration of an agent run, we eliminated most of that overhead.

Latency is not a number you chase at the end. It's a constraint you hold from the first line of the design.

The optimizations that mattered

Not every change moved the needle. These four are the ones that, together, took us from 240ms to 82ms at the median:

  • Warm execution contexts that persist across an agent's tool calls, removing per-step scheduling cost.
  • Speculative prefetching of likely-needed context before the model finishes its current step.
  • A rewritten serialization path that avoids redundant JSON encoding between internal services.
  • Region-aware routing that keeps inference physically close to the calling application.

What it unlocks

Sub-100ms responses change what's possible. Real-time voice agents, inline code completion, and interactive support flows all become viable when the model's latency drops below the threshold of perceptible delay. The work was never about the number — it was about the products the number makes possible.