Observability Setup

Wire AllRoutes to OpenTelemetry, Langfuse, Helicone, and Datadog for end-to-end tracing of every AI request.

Overview

You can answer the basic questions ("what did it cost? how slow was it?") from the AllRoutes Analytics dashboard without integrating anything. For deeper investigation -- which prompt produced this hallucination? why did p99 latency double last Tuesday? -- you want full traces in a dedicated observability tool.

This guide walks through setup for the four most common destinations:

OpenTelemetry (OTEL) -- vendor-neutral, exports to anything (Jaeger, Tempo, Honeycomb, etc.)
Langfuse -- LLM-native tracing with prompts, responses, evaluations
Helicone -- request/response logging with cost analytics
Datadog -- APM correlation with the rest of your services

All of these are configured via Broadcast destinations -- the gateway streams traces to them; you don't add SDKs or wrap calls in your application.

OpenTelemetry

What You Get

OTEL spans for every chat completion, embedding, and tool call, with these attributes:

Attribute	Type	Example
`gen_ai.system`	string	`openai`, `anthropic`, `google`
`gen_ai.request.model`	string	`gpt-4o`
`gen_ai.response.model`	string	`gpt-4o-2024-08-06`
`gen_ai.request.temperature`	float	`0.7`
`gen_ai.usage.input_tokens`	int	`28`
`gen_ai.usage.output_tokens`	int	`156`
`gen_ai.response.finish_reasons`	array	`["stop"]`
`allroutes.cost_usd`	float	`0.0032`
`allroutes.cache.status`	string	`hit` / `miss`
`allroutes.fallback_count`	int	`0`

These follow the OpenTelemetry GenAI semantic conventions.

Setup

curl -X POST https://api.allroutes.ai/v1/broadcast/destinations \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -H "Content-Type: application/json" \
  -d '{
    "type": "otel",
    "config": {
      "endpoint": "https://otel-collector.your-org.com:4318/v1/traces",
      "headers": {"x-api-key": "your-collector-key"},
      "protocol": "http/protobuf"
    },
    "sampling_rate": 1.0,
    "privacy_mode": false
  }'

Supported protocols: http/protobuf, http/json, grpc. Use any OTEL-compatible backend (Honeycomb, Grafana Tempo, Lightstep, AWS X-Ray, GCP Cloud Trace).

Correlating with Your Application Spans

If your application is already instrumented with OTEL, propagate the trace context via the traceparent header:

from opentelemetry.trace import get_current_span
from opentelemetry.propagate import inject

headers = {}
inject(headers)  # populates traceparent

response = client.chat.completions.create(
    model="gpt-4o",
    messages=msgs,
    extra_headers=headers,
)

The gateway picks up traceparent and stitches the AllRoutes span as a child of your application span -- you'll see the full call tree in your observability tool.

Langfuse

What You Get

Langfuse stores prompts, responses, scores, and evaluations. Use it when you care about LLM-specific concerns: prompt versions, output evaluation, conversation replay.

Setup

Get your Langfuse keys from cloud.langfuse.com (or your self-hosted instance) under Settings > API Keys
Add the broadcast destination:

curl -X POST https://api.allroutes.ai/v1/broadcast/destinations \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -H "Content-Type: application/json" \
  -d '{
    "type": "langfuse",
    "config": {
      "public_key": "pk-lf-...",
      "secret_key": "sk-lf-...",
      "host": "https://cloud.langfuse.com"
    },
    "sampling_rate": 1.0,
    "privacy_mode": false
  }'

Tag requests with metadata that Langfuse will surface as traces:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=msgs,
    user="usr_123",
    session_id="sess_456",
    metadata={
        "langfuse_tags": ["production", "support-bot"],
        "experiment": "prompt-v3",
    },
)

The session_id becomes a Langfuse session (groups multi-turn conversations); user becomes the Langfuse user; metadata.langfuse_tags becomes filterable tags.

Scoring and Evaluation

Langfuse evaluators (LLM-as-judge, custom Python functions) run automatically on incoming traces. Configure them in the Langfuse UI under Evaluators -- no AllRoutes config needed.

Helicone

What You Get

Helicone is a request/response log viewer with cost and latency analytics, similar to AllRoutes' built-in analytics but with a different UI and stronger session/conversation views.

Setup

curl -X POST https://api.allroutes.ai/v1/broadcast/destinations \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -H "Content-Type: application/json" \
  -d '{
    "type": "helicone",
    "config": {
      "api_key": "sk-helicone-..."
    },
    "sampling_rate": 1.0
  }'

Tag with Helicone-style metadata:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=msgs,
    metadata={
        "Helicone-Property-Feature": "support-bot",
        "Helicone-Property-Tenant": "acme-corp",
    },
)

Properties prefixed with Helicone-Property- become filterable in the Helicone dashboard.

Datadog

What You Get

If your services are already in Datadog APM, broadcasting AllRoutes traces lets you see AI calls inside the same flame graphs as your DB queries and HTTP fetches.

Setup

curl -X POST https://api.allroutes.ai/v1/broadcast/destinations \
  -H "Authorization: Bearer allroutes_mgmt_..." \
  -H "Content-Type: application/json" \
  -d '{
    "type": "datadog",
    "config": {
      "api_key": "DD_API_KEY",
      "site": "datadoghq.com",
      "service": "allroutes-gateway",
      "env": "production"
    },
    "sampling_rate": 0.1
  }'

Use a lower sampling_rate for Datadog -- AI traces tend to be high-volume and Datadog bills per ingested span. 0.1 (10%) is typical.

Correlating with APM

Same as OTEL: propagate the Datadog trace headers (x-datadog-trace-id, x-datadog-parent-id) to AllRoutes:

from ddtrace import tracer

with tracer.trace("ai.call") as span:
    headers = {
        "x-datadog-trace-id": str(span.trace_id),
        "x-datadog-parent-id": str(span.span_id),
    }
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=msgs,
        extra_headers=headers,
    )

Combining Destinations

You can have all four active simultaneously, each with its own sampling rate:

Destination	Sampling	Cost Profile	Use For
Langfuse	100%	Per-trace, cheap	Prompt debugging, evals
OTEL → Tempo	100%	Self-hosted, free	Engineering investigation
Datadog	10%	Per-span, expensive	APM correlation
BigQuery	100%	Per-byte, cheap at scale	Long-term analytics

Configure each as a separate destination -- they don't interfere with one another.

Privacy Considerations

Broadcast payloads include prompt and response content by default. To strip them:

{"privacy_mode": true}

This sends only metadata (model, tokens, cost, latency) -- useful for HIPAA, GDPR, and customers under NDA. Combine with Guardrails ZDR for the strictest mode.

Troubleshooting

Symptom	Cause	Fix
No spans in destination	Wrong endpoint or auth	Test config with the destination's "send test event" feature
Spans appear without parent	Missing `traceparent` header propagation	Ensure your app injects context
Datadog bill spiked	Sampling too high	Drop `sampling_rate` to 0.1 or lower
Langfuse traces missing prompts	`privacy_mode: true` enabled	Set `privacy_mode: false` and re-deploy
OTEL spans missing tokens	Wrong protocol version	Use `http/protobuf` (most compatible)

Overview

OpenTelemetry

What You Get

Setup

Correlating with Your Application Spans

Langfuse

What You Get

Setup

Scoring and Evaluation

Helicone

What You Get

Setup

Datadog

What You Get

Setup

Correlating with APM

Combining Destinations

Privacy Considerations

Troubleshooting

See Also