$340k/year saved by routing background jobs to the cheapest healthy model

Challenge

Northway Systems classifies and summarises shipment-tracking events for freight customers — a quiet, high-volume workload that runs ~3.1M LLM turns a month. They'd standardised on a top-tier model because it was the one their app team had wired up first, but most of those calls were short, low-stakes classifications that didn't benefit from a flagship model.

The bill was real, the latency wasn't — and the team had no easy way to A/B different models on the workload without rewriting half the worker fleet.

Solution

Northway routed their background workers through AllRoutes with :floor — route to the cheapest provider currently meeting the quality bar. They added a routing rule constraining the candidate set to their three preferred providers and a max-tokens cap.

For the customer-facing dashboard explanations they kept the premium model (with :nitro for speed). One client, two policies.

Result

• 41% reduction in per-month LLM spend in the first full billing cycle — $340k/year run-rate.
• Background latency stayed inside SLO — the floor router only fails over to a cheaper model when its measured quality on similar prompts is within 2% of the baseline.
• Zero worker-code changes to A/B the next pricing change — Northway tweaks a routing rule in the dashboard.

The most-quoted internal line from the rollout: "We didn't need GPT-4o for shipment summaries. AllRoutes figured that out before we did."