Every LLM provider goes down. Anthropic gets rate-limited mid-incident; OpenAI returns 503s during a model rollout; DeepSeek's network has a bad afternoon. If your product makes an LLM call on the user-visible path, you eventually face a choice: surface the outage to the user, or have a backup plan.
The backup plan that actually works at MVP scale is cross-provider failover — when your primary provider returns a retryable error, transparently retry against a different provider. It's tempting to bolt this on as rescue ProviderError; try_other_provider; end and call it done. But there are two traps in that naive sketch that bite you the first time something goes wrong, and they're both about observability.
This post is what we landed on, and the pieces I'd skip if I were doing it again.
The naive approach burns observability
Here's the rescue-and-retry version:
def call_llm(prompt)
AnthropicAdapter.new.complete(prompt)
rescue Anthropic::Errors::RateLimitError, Anthropic::Errors::APIError
OpenAIAdapter.new.complete(prompt)
end
This works in the happy-failover case. Anthropic 429s, OpenAI succeeds, the user gets their response, you go to bed.
But three days later, you're trying to figure out why your AI bill spiked. You open the usage log and see one row per request — provider always the second one half the time. There's no record that the first provider failed. You can't tell:
- Whether failover is firing on 1% of requests or 40%.
- Which feature is producing failovers (parsing? ATS? tailoring?).
- Whether failures are clustered in a single 10-minute window (a real outage) or smeared across the day (a degrading provider).
- Whether a "successful" call took two attempts to land — relevant for latency budgets.
You wrote the failover to improve UX. You did. But you also stopped seeing the underlying problem. The slow-rotting primary that failover is silently saving you from never appears in any chart.
Log every attempt, share a request_id
Once you accept that failover is itself an observability event, the design follows.
Every attempt — the one that failed AND the one that succeeded — writes its own usage-log row. Both rows share a request_id (a UUID minted at the start of the logical call) and an attempt index (1, 2, 3...). Now a failover-driven request looks like this:
| id | request_id | provider | model | attempt | success | error_message |
|---|---|---|---|---|---|---|
| 412 | a3f1... |
anthropic | claude-sonnet-4-6 | 1 | false | RateLimitError: 429 |
| 413 | a3f1... |
openai | gpt-4o | 2 | true | nil |
A glance at row 412 tells you the primary failed. The matching request_id on 413 tells you the user still got their response. The attempt: 2 flag tells the latency analysis which row was on the user's clock.
Group by (provider, success) and you have the per-provider failure rate. Filter by attempt: 2 AND success: true and you have the rate at which failover is silently saving you. Group by created_at (5-minute buckets) and you can see incident windows.
In our schema, this is just one extra column (metadata is JSONB, holds the request_id, attempt, attempts_total) and the discipline of calling the logger per attempt. No new table, no separate "failover events" model. The same query you already run for "which feature is most expensive this week" now also tells you which provider had the worst week.
Don't failover on the wrong errors
The second trap is failover on errors that won't be saved by trying another provider.
Provider gem error hierarchies fall into roughly four categories:
- Rate limit (429) — retryable. Try again later, or try elsewhere.
- Service error (5xx, network, timeout) — retryable. Same.
- Bad request (400) — NOT retryable. The same input will fail the same way on the next provider, and you'll burn the second budget for the privilege.
- Configuration / auth (401, 403) — NOT retryable. If your Anthropic key is wrong, your OpenAI key probably has its own issues; failover is the wrong response.
The naive rescue above caught the first two correctly but, depending on the gem, also caught the bad-request case (Anthropic's BadRequestError is a subclass of APIError). When a paying customer's prompt was malformed — say, exceeded the model's context window — we burned an OpenAI call to fail the same way, then logged a single "OpenAI 400" row. The customer saw the same error message, just delivered slower.
The fix is two parallel error categorisations:
RETRYABLE = [Llm::Errors::RateLimitError, Llm::Errors::ServiceError]
NON_RETRYABLE = [Llm::Errors::BadRequestError, Llm::Errors::ConfigurationError]
Each adapter translates its underlying gem's exceptions into one of these four abstract categories. Service code rescues only the abstract classes. The dispatcher (we call ours Llm::Client) checks which list the error falls into:
- Retryable → log the failed attempt, try the next provider in the chain.
- Non-retryable → log the failed attempt, raise immediately. No failover.
The "log the failed attempt" half is the same in both cases — that's how we know later that a 400 happened. The "try the next provider" half is what changes.
We also tag the row with failover_eligible: true | false, derived directly from which list the error fell into. When auditing whether failover is firing for the right reasons, this lets you ask "did we ever try to failover a 400 error?" — the answer should be no. If it isn't, your error categorisation drifted and you want to know.
When the whole chain fails, that's an alert
There's one more state worth distinguishing: the failover chain is exhausted and every provider failed on retryable errors. That's not "one provider had a bad minute" — it's a multi-provider outage, or a network-level problem on your side.
Llm::Client fires a deduped admin alert (15-minute window per feature × error_class tuple) only at this point. Single-provider exhaustion isn't an outage if failover saves us. Whole-chain exhaustion is the signal you actually want to wake someone up for.
The dedup window matters. A persistent multi-provider issue would otherwise generate one alert email per failed user request — at which point the alert becomes noise and gets filtered out. Cap it at one email per feature × error_class per 15 minutes and the inbox stays useful.
Two patterns I'd skip if doing this again
Don't put failover in the service. Earlier drafts had each of our three LLM-using services (parsing, tailoring, ATS) wrap their own try/catch with the failover logic. The duplication wasn't the worst part — the worst part was that each service evolved a slightly different definition of "retryable." The dispatcher pattern, where one Llm::Client owns the whole question, was cheap to extract and made the right thing the easy thing.
Don't muddle in-provider retry with cross-provider failover. They're solving different problems. In-provider retry exists for "the network blipped, try once more" — fast, cheap, often successful. Cross-provider failover exists for "this provider is having a bad afternoon, route around it" — more expensive, requires the second provider to actually exist. Wrapping both into one variable-attempts loop muddied the alerting (does an admin alert fire after 2 retries against Anthropic, or only after Anthropic + OpenAI both fail?). Splitting them — adapters retry their own provider 1× with backoff; the dispatcher iterates the chain — made the alerting boundary obvious.
What this unlocks
The first reward is reliability: a single-provider outage stops being a user-visible event. The second is more interesting — once every attempt is logged, you can run the per-provider failure-rate query as a normal admin dashboard panel. You can spot a provider drifting before the on-call person feels it. You can A/B a new provider against the existing primary by adding it as a failover entry, watching the failover-driven rows for a week, and only promoting it if the failure rate is healthy.
That last point is the real upside. Failover for resilience is the obvious value. Failover as a deployable observability tool — that's the one I didn't expect, and the one I'd build for first if I were doing it again.