Failover & Circuit Breaking
Each provider has its own in-memory circuit, per gateway process. Five consecutive failures opens it; a request against an open provider is skipped in favor of the next candidate, so a known-down provider doesn't eat a full timeout on every request.
Thresholds
| Constant | Value | Meaning |
|---|---|---|
FAILURE_THRESHOLD | 5 | Consecutive failures before a provider's circuit opens. |
COOLDOWN_MS | 30_000 | Time after opening before the circuit is treated as half-open. |
A success at any point resets consecutiveFailures to 0 and clears the open timestamp. A failure while already open does not reset the cooldown timer — it doesn't extend how long the provider stays skipped.
half-open is not specially filtered by dispatch — only open circuits are excluded from the candidate list. Once the cooldown elapses, the very next request that would have used that provider is the probe; if it succeeds, the circuit closes, and if it fails, consecutiveFailures keeps climbing and the circuit re-opens for another 30 seconds. What counts as a retryable failure
A failed attempt only moves on to the next candidate — and only counts against the circuit — if the error is one of:
- A
ProviderRequestErrorwith an HTTP status>= 500. - A timeout (
DOMExceptionnamedTimeoutError). - A network-level failure (a plain
TypeError, e.g. DNS failure or connection refused).
A 4xx from the provider is not retried: the request itself was malformed for that provider, and no other provider would accept it either. That failure is returned to the client immediately as-is, without touching the circuit.
Non-streaming vs. streaming retry windows
| Timeout | Failover window | |
|---|---|---|
| Non-streaming | 60 seconds per attempt | Any retryable failure, for every candidate in the resolved order. |
| Streaming | 30-second idle timeout, reset on every chunk received | Only before the first chunk is produced — see Request Lifecycle . |
If every candidate for a model is open at once, the gateway tries the original declared order anyway rather than failing the request on its own bookkeeping — a stuck circuit breaker should never make things worse than having no circuit breaker at all.
Exhausting all candidates
If every attempt in the resolved order fails:
- More than one attempt was made →
502 all_providers_failed. - Only one candidate existed and it failed →
502 provider_error.
Multi-instance sync
Circuit state and latency averages are process-local for reads (checking currentState() never does I/O on the request path), but the gateway is expected to run as more than one instance. A Nitro plugin (plugins/state-sync.ts) runs every 5 seconds:
- Push: upsert this instance's locally-known circuit/latency state into
provider_circuit_state/provider_latency_state. - Pull: read every instance's rows back and merge — for circuit state, keep whichever of (local, remote) has the higher
consecutiveFailures.
That merge policy is deliberately fail-open-to-caution: one instance's bad experience with a provider makes every instance cautious about it, but each instance still has to independently observe its own successes to fully close the circuit again. Sync failures are logged and swallowed — they never crash the process or block a request.