Smart Routing

Overview

AllRoutes Smart Routing automatically selects the optimal provider and model for each request based on your chosen strategy. It handles failovers, load balancing, and cost optimization transparently.

Routing Strategies

Set the routing strategy via the provider field in your request:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello!"}],
  "provider": {
    "strategy": "cost"
  }
}

Available Strategies

Strategy	Behavior
`cost`	Route to the cheapest available provider for the requested model
`latency`	Route to the provider with the lowest measured latency
`throughput`	Route to the provider with the highest available throughput
`auto`	Balanced optimization across cost, latency, and reliability
`free`	Prefer free model variants when available (community-hosted, rate-limited)

Example: Latency-Optimized

curl https://api.allroutes.ai/v1/chat/completions \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "provider": {
      "strategy": "latency",
      "allow_fallbacks": true
    }
  }'

Model Fallbacks

Use the models array to define an ordered fallback chain. AllRoutes tries each model in sequence, moving to the next if the current one fails (rate limit, outage, timeout):

{
  "models": ["gpt-4o", "claude-sonnet-4-20250514", "gemini-2.0-flash"],
  "messages": [{"role": "user", "content": "Summarize this document."}]
}

When models is provided, the model field is ignored. The first available model in the array is used.

Fallback Triggers

A fallback is triggered when the current model returns:

429 -- Rate limit exceeded
500/502/503 -- Provider error or outage
Timeout -- No response within the configured timeout
Context overflow -- Input exceeds the model's context window

Provider Preferences

Fine-tune routing with the provider object:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello!"}],
  "provider": {
    "order": ["azure", "openai"],
    "allow_fallbacks": true,
    "require_parameters": true,
    "data_collection": "deny"
  }
}

Field	Type	Description
`strategy`	string	Routing strategy (`cost`, `latency`, `throughput`, `auto`, `free`)
`order`	array	Preferred provider order (e.g., `["azure", "openai"]`)
`allow_fallbacks`	boolean	Allow falling back to other providers (default: `true`)
`require_parameters`	boolean	Only route to providers that support all request parameters
`data_collection`	string	Data collection preference: `"allow"` or `"deny"`

Custom Routing Rules

Define organization-level routing rules in the dashboard to enforce policies:

Region routing -- route to providers in specific regions (US, EU, Asia)
Model pinning -- map model aliases to specific versions
Provider blocking -- exclude specific providers from routing
Cost caps -- set per-request cost limits that trigger cheaper alternatives

A/B Testing

Split traffic between models to compare quality and cost:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello!"}],
  "provider": {
    "strategy": "auto",
    "ab_test": {
      "variants": [
        {"model": "gpt-4o", "weight": 50},
        {"model": "claude-sonnet-4-20250514", "weight": 50}
      ]
    }
  }
}

Results are tracked in the dashboard with per-variant metrics for latency, cost, and quality scores.

BYOK Priority Routing

When you have Bring Your Own Key (BYOK) provider keys configured, AllRoutes prioritizes them:

Your provider key is tried first (0% commission)
If your key fails or is rate-limited, AllRoutes keys are used as fallback
Platform fee applies only when AllRoutes keys are used

This gives you the best of both worlds: zero-cost routing when your keys work, with automatic fallback for reliability.

Circuit Breaker

AllRoutes implements a circuit breaker pattern for each provider:

Closed (normal) -- requests flow through normally
Open (tripped) -- provider is temporarily removed from routing after repeated failures
Half-Open (probing) -- periodic test requests check if the provider has recovered

The circuit breaker prevents cascading failures and ensures requests are routed to healthy providers. It operates automatically and requires no configuration.

Response Headers

When route_explain: true is set in the request, the response includes detailed routing information:

Header	Description
`X-AllRoutes-Provider`	Provider that served the request
`X-AllRoutes-Model`	Resolved model identifier
`X-AllRoutes-Route-Strategy`	Routing strategy used
`X-AllRoutes-Fallback-Count`	Number of fallback attempts before success
`X-AllRoutes-Latency-Ms`	Total request latency

SDK Examples

Python

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    provider={"strategy": "latency", "allow_fallbacks": True},
)

Node.js

const completion = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
  provider: { strategy: "cost", order: ["azure", "openai"] },
});