62% drop in p99 chat latency with cost-aware routing

Challenge

Volta Labs ships a developer-facing chat product on top of three LLM providers. When their primary provider's p99 spiked above 8 seconds, every Volta customer felt it — and the team's on-call had built a fragile mesh of per-provider retries, circuit breakers, and per-region failovers that lived in three different repos.

The runtime cost of a single chat turn varied 4× between providers on identical prompts. There was no good way to choose at request time which provider would land cheapest and fastest for that specific query shape.

Solution

Volta moved every chat call behind AllRoutes' OpenAI-compatible endpoint and turned on :nitro routing for the public chat surface — routes to whichever provider is currently fastest for the model family — plus :floor routing for background summarisation jobs where latency didn't matter and cost did.

client = AllRoutes(api_key=os.environ["ALLROUTES_KEY"])
response = client.chat.completions.create(
  model="claude-sonnet-4-6:nitro",
  messages=[...],
)

That was the whole change. The router handles provider health, retries, fallback, and per-request cost/latency optimisation; Volta deleted the entire retry mesh from their app code.

Result

• 62% drop in p99 chat latency in the first two weeks.
• $8,400/month saved on background summarisation by routing to the cheapest healthy provider per request.
• 3 weeks of engineer time reclaimed that had been earmarked for "fix the retry logic, again."

The on-call rotation went from "always on edge for provider incidents" to "rarely paged" — Volta now treats LLM provider outages the way they treat cloud zone outages: AllRoutes routes around them.