Streaming

Server-Sent Events protocol for chat completions, including chunk format, [DONE] handling, and stream_options.include_usage.

Overview

AllRoutes streams chat completions over Server-Sent Events (SSE). When you set stream: true on a chat completion request, the gateway flushes incremental tokens to your client as they arrive from the upstream provider, with a small (~5-15ms) gateway-side overhead for cache and guardrail checks.

The format is byte-compatible with OpenAI's streaming protocol, so any OpenAI-compatible client works out of the box.

Enabling Streaming

curl https://api.allroutes.ai/v1/chat/completions \
  -H "Authorization: Bearer allroutes_sk_..." \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Count to 5."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The response uses Content-Type: text/event-stream, Cache-Control: no-cache, and Connection: keep-alive.

Chunk Format

Each chunk is a single SSE event prefixed with data: and terminated by a blank line. The payload is a JSON object with the same envelope as a non-streamed response, but with a delta instead of message:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1714000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1714000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1714000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":", two"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1714000000,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1714000000,"model":"gpt-4o","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16}}

data: [DONE]

Chunk Fields

Field	Type	Description
`id`	string	Stable across all chunks for one request
`object`	string	Always `chat.completion.chunk`
`model`	string	Resolved model identifier
`choices[].delta.role`	string	Sent only on the first content chunk
`choices[].delta.content`	string	Token fragment to append to the assistant message
`choices[].delta.tool_calls`	array	Incremental tool-call construction
`choices[].delta.reasoning`	string	Reasoning/thinking fragment (Anthropic, DeepSeek)
`choices[].finish_reason`	string	`null` until the final chunk; then `stop`, `length`, `tool_calls`, `content_filter`
`usage`	object	Final token counts (only when `stream_options.include_usage: true`)

The `[DONE]` Sentinel

The literal string data: [DONE] is sent after the last JSON chunk. Treat it as the end-of-stream signal -- do not attempt to JSON-parse it. Most OpenAI-compatible parsers handle this automatically.

Stream Options

Pass stream_options to control extra metadata:

Field	Type	Default	Description
`include_usage`	boolean	`false`	Emit a final chunk with `usage` token counts before `[DONE]`

When include_usage: true, the final pre-[DONE] chunk has an empty choices: [] and a populated usage object. This is the only reliable way to get token counts from a streamed response.

Tool Call Streaming

Tool calls arrive incrementally. Each chunk's delta.tool_calls[] carries an index (the tool call number) and a partial function.arguments string. Concatenate by index:

delta.tool_calls = [{"index": 0, "id": "call_abc", "function": {"name": "get_weather", "arguments": ""}}]
delta.tool_calls = [{"index": 0, "function": {"arguments": "{\"loc"}}]
delta.tool_calls = [{"index": 0, "function": {"arguments": "ation\":"}}]
delta.tool_calls = [{"index": 0, "function": {"arguments": " \"NYC\"}"}}]

The full arguments string is {"location": "NYC"}. Only parse JSON once finish_reason: "tool_calls" is observed.

SDK Examples

Python

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Tell a story."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n\nTotal tokens: {chunk.usage.total_tokens}")

Node.js

const stream = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Tell a story." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
  if (chunk.usage) console.log(`\nTokens: ${chunk.usage.total_tokens}`);
}

Raw fetch (browser/Edge)

const res = await fetch("https://api.allroutes.ai/v1/chat/completions", {
  method: "POST",
  headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "gpt-4o", messages, stream: true }),
});

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6).trim();
    if (data === "[DONE]") return;
    const chunk = JSON.parse(data);
    const delta = chunk.choices?.[0]?.delta?.content ?? "";
    if (delta) console.log(delta);
  }
}

Heartbeats and Reconnection

For long-running streams, AllRoutes emits SSE comment frames (: keepalive) every 15 seconds to prevent intermediate proxies (Cloudflare, AWS ALB) from closing the connection. Comment frames are silently ignored by SSE parsers.

If your client disconnects mid-stream, the upstream provider call is not automatically resumed -- you must issue a new request. Use idempotency keys via the metadata.idempotency_key field if duplicate-call risk matters.

Troubleshooting

Symptom	Likely Cause	Fix
Stream returns one big chunk	Buffering proxy in front of your client	Set `Accept: text/event-stream` and disable proxy buffering (e.g., nginx `proxy_buffering off`)
`usage` always missing	`stream_options.include_usage` not set	Pass `{"include_usage": true}`
Random `499` mid-stream	Client disconnected (timeout, abort)	Increase client timeout or add heartbeats
Stream hangs after first chunk	Provider rate-limit	Check the `X-AllRoutes-Provider` header and switch via `provider.order`