Model routing and fallback: your app's real reliability layer

Every team we work with eventually hits the same 2am page: a single LLM provider has a bad hour, and every feature that calls it goes down with it. Not because the model is wrong — because the call to it has no fallback path. Model routing isn’t an optimization you add later. It’s the reliability layer your product is missing.

The failure modes aren’t interchangeable

Treating “the LLM call failed” as one bucket is the first mistake. Different failures need different responses:

• 429 (rate limited) — you hit your own cap. The right move is often a same-provider downgrade to a cheaper, faster model, not a retry against the same limit.
• 5xx / timeout — the provider’s infra is unhealthy. Retrying the same endpoint wastes latency. Rotate to a different provider entirely.
• Guardrail or moderation block — the request itself is the problem. Retrying anywhere will fail the same way; this needs a different code path, not a fallback model.
• Quality-floor miss — the response came back but failed a cheap validator (empty output, malformed JSON, obvious refusal). This is where a stronger model as fallback, not a cheaper one, is correct.

Conflating these into one generic “retry 3x then give up” handler is why routing logic that looks reasonable in a design doc falls over in production.

A composition that holds up

The pattern that has proven durable across client stacks is layered, in this order:

flowchart LR
    A[Request] --> B[Primary model]
    B -->|429| C[Downgrade: cheaper model, same provider]
    B -->|5xx / timeout| D[Rotate: secondary provider]
    B -->|quality-floor miss| E[Escalate: stronger model]
    C --> F{Success?}
    D --> F
    E --> F
    F -->|no, chain exhausted| G[Semantic cache hit]
    G -->|miss| H[Degraded response to user]
    F -->|yes| I[Return response]

1. Retry the primary once or twice with jitter — most transient failures resolve here, and burning your fallback budget on noise is wasteful.
2. On retry exhaustion, rotate providers, not just models. A second OpenAI model doesn’t help if OpenAI’s infra is down.
3. On full chain exhaustion, serve a semantic cache hit if you have one for a similar-enough request. Stale-but-correct beats an error page.
4. On cache miss, degrade the UI honestly — “we’re having trouble generating this right now, try again” beats a silent hang or a raw stack trace.

Each hop should be logged with which route it took and why. If you can’t answer “how often are we falling back, and to what” from a dashboard, you don’t have a routing strategy — you have a retry loop and a hope.

Don’t route on vibes

Two things make routing decisions trustworthy instead of guesswork:

• A held-out quality floor. Define, per task, the minimum acceptable output — schema validity, length bounds, a cheap LLM-judge score — and use it to decide whether a fallback response is actually good enough to return, not just non-empty.
• Idempotency keys on every request that can retry or rotate. Without them, a slow-but-eventually-successful primary call and a fallback call can both land — duplicate side effects, double-charged users, two emails sent. This bites teams almost exactly once, expensively.

Build vs. buy

For teams making more than a couple of LLM calls in the critical path, a gateway (LiteLLM, Portkey, OpenRouter, or an internal equivalent) is worth adopting over hand-rolled routing. You get provider rotation, circuit breakers, and consistent OpenTelemetry GenAI spans for free, and your application code calls one interface regardless of what’s behind it that day. Hand-roll it only if your routing logic is genuinely simple — one provider, one fallback model — because a homegrown gateway that only gets exercised during an actual outage is a gateway you’ve never really tested.

What to instrument before you need it

• Route taken per request (primary / downgrade / rotate / cache / degraded), as a metric, not just a log line.
• Fallback rate by failure type, so a spike in 429-driven downgrades reads differently from a spike in 5xx-driven rotations.
• Cost delta between primary and fallback paths — fallback logic that quietly triples your spend during an incident is its own outage.
• Time-to-first-fallback-hop, so you know your circuit breakers are actually tripping fast enough to matter.

The goal isn’t zero failures — providers will have bad days no matter what you do. The goal is that your users never notice, and your on-call engineer finds out from a dashboard instead of a support ticket.