Model Groups

Alias multiple concrete models behind a single virtual identifier with weighted routing, fallbacks, and per-group cost controls.

Overview

A model group is a virtual model identifier that resolves to one of several concrete models at request time. Use them to:

Decouple your application code from specific model SKUs ("call cheap-fast-chat" instead of "call gpt-4o-mini")
Run weighted A/B tests across providers without changing client code
Route by tenant, environment, or feature flag without redeploying
Enforce per-group cost caps independently of per-key budgets

When your application sends model: "cheap-fast-chat", the gateway resolves the group, picks a member according to the group's strategy, and forwards the request. Token counting, cost reporting, and analytics roll up under the group identifier so you can swap members without losing historical comparability.

Group vs. Preset vs. Fallback Array

Feature	Best For
Presets	Bundle model + parameters + system prompt + plugins
`models` fallback array	Per-request, ordered failover
Model groups	Server-side aliasing, weighted routing, central control

You can combine all three: a preset can reference a model group, and a model group's strategy can specify a fallback chain.

Creating a Group

POST https://api.allroutes.ai/v1/model-groups

Request Body

Field	Type	Required	Description
`name`	string	Yes	Group identifier (referenced as `model` in requests)
`members`	array	Yes	Concrete model identifiers, optionally with `weight` and `provider`
`strategy`	string	No	`round_robin`, `weighted`, `cost`, `latency`, `failover` (default: `weighted`)
`daily_budget_usd`	float	No	Cap on aggregate daily spend across the group
`description`	string	No	Human-readable description

Example

curl -X POST https://api.allroutes.ai/v1/model-groups \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cheap-fast-chat",
    "strategy": "weighted",
    "members": [
      {"model": "gpt-4o-mini", "weight": 60},
      {"model": "claude-haiku-3-5", "weight": 30},
      {"model": "gemini-2.0-flash", "weight": 10}
    ],
    "daily_budget_usd": 100.00,
    "description": "Production tier-1 chat: cheap, fast, mostly OpenAI"
  }'

Response

{
  "id": "mg_abc123",
  "name": "cheap-fast-chat",
  "strategy": "weighted",
  "members": [...],
  "daily_budget_usd": 100.00,
  "spent_today_usd": 0.00,
  "created_at": "2026-04-05T10:00:00Z"
}

Using a Group

Reference the group's name as if it were a regular model:

response = client.chat.completions.create(
    model="cheap-fast-chat",
    messages=[{"role": "user", "content": "Hello!"}],
)

# The actual model picked is in response.model
print(response.model)  # e.g., "gpt-4o-mini"

The response includes the resolved model in response.model and the group name in the X-AllRoutes-Model-Group header.

Strategies

Strategy	Behavior
`round_robin`	Cycle through members evenly
`weighted`	Pick a member by weight (members sum doesn't need to equal 100)
`cost`	Always pick the cheapest currently-available member
`latency`	Always pick the lowest-p95-latency member (auto-updated every minute)
`failover`	Always pick the first member; fall through on rate-limit or error

For cost and latency, AllRoutes maintains rolling 5-minute windows to avoid pinning to one member after a transient blip.

Per-Member Provider Override

Force a member to use a specific provider:

{
  "members": [
    {"model": "claude-sonnet-4-20250514", "provider": "anthropic", "weight": 50},
    {"model": "claude-sonnet-4-20250514", "provider": "bedrock", "weight": 50}
  ]
}

This is useful for running the same model across providers as a region or pricing hedge.

Updating, Listing, Deleting

# List
curl https://api.allroutes.ai/v1/model-groups \
  -H "Authorization: Bearer allroutes_mgmt_..."

# Get details
curl https://api.allroutes.ai/v1/model-groups/mg_abc123 \
  -H "Authorization: Bearer allroutes_mgmt_..."

# Update weights (live; no redeploy needed)
curl -X PATCH https://api.allroutes.ai/v1/model-groups/mg_abc123 \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -d '{"members": [{"model": "gpt-4o-mini", "weight": 100}]}'

# Delete
curl -X DELETE https://api.allroutes.ai/v1/model-groups/mg_abc123 \
  -H "Authorization: Bearer allroutes_mgmt_..."

Updating weights is the recommended way to roll out a new model: start at 5%, watch quality and cost in Analytics, and ramp to 100% over hours or days.

Group Budget Enforcement

When daily_budget_usd is set, the gateway tracks spend across all group calls. When the cap is reached:

Subsequent requests return 402 Payment Required with error.code: "model_group_budget_exhausted"
The cap resets at midnight UTC
A budget.threshold and budget.exhausted webhook fires

Group budgets are independent of per-key budgets -- both can apply to the same request.

A/B Testing Pattern

Combine weighted strategy with Analytics custom metadata to evaluate quality:

response = client.chat.completions.create(
    model="cheap-fast-chat",
    messages=msgs,
    metadata={"experiment": "tier1-2026-q2"},
)

In the dashboard, filter by metadata.experiment = "tier1-2026-q2" and group by model to see per-member latency, cost, healing rate, and refusal rate side by side.

Best Practices

Pick descriptive group names -- cheap-fast-chat, vision-best, coding-tier. Avoid version suffixes; use weight rolls instead.
Hedge across providers -- list the same model from two providers (Anthropic + Bedrock for Claude) for outage resilience.
Use failover for production -- weighted is for experiments; failover for the safe path.
Set group budgets above key budgets -- group budget caps the worst-case spend across all keys using the group.
Review monthly -- prune members that consistently underperform on cost or quality.

Troubleshooting

Symptom	Cause	Fix
`404: model not found`	Group name typo or deleted	Check `/v1/model-groups` listing
Always picks one member	Weights heavily skewed or `failover` strategy	Confirm strategy and weight distribution
Cost vs. expected	A member is more expensive than baseline	Switch to `cost` strategy or remove the member
`model_group_budget_exhausted`	Group daily budget hit	Increase budget or reduce weights on expensive members