Request Lifecycle

A chat completion is authenticated, checked against quota, routed to a candidate provider chain, and logged — without any of that bookkeeping blocking the response the client sees.

Step by step#

1. Request ID (middleware/01.request-id.ts) — every incoming request is stamped with a server-generated Bun.randomUUIDv7(), stored in a per-Request context (see Security), used to correlate structured log lines.
2. Auth (middleware/02.auth.ts, /v1/** only) — the authorization: Bearer <key> header is required. Missing → 401 missing_api_key. The key is hashed and looked up; no match, or the owning application is disabled → 401 invalid_api_key. On success, the resolved Application row is attached to the request context.
3. Rate limit (middleware/03.rate-limit.ts, /v1/** only) — if the application has a rateLimitPerMinute, a fixed 60-second window counter is incremented; exceeding it → 429 rate_limit_exceeded. See Rate Limits & Quotas.
4. Body validation — the JSON body is parsed and validated against chatCompletionRequestSchema (a Valibot strictObject, so unrecognized fields like tools are rejected outright rather than silently ignored). Failure → 400 invalid_request.
5. Model permission (utils/policy.ts) — if the application has an allowedModels list, the raw model string from the request must be in it (checked before alias resolution). Violation → 403 model_not_permitted.
6. Quota reservation (utils/quota.ts) — if the application has a monthlyTokenQuota, an upper-bound token estimate (min(max_tokens ?? 4096, 4096)) is atomically reserved against the current month's counter in the same conditional database write used to check it, so concurrent requests can't all pass a stale check. Exceeded → 429 quota_exceeded. The reservation is refunded on failure and reconciled down to real usage once the provider responds.
7. Dispatch — resolve the model to a candidate provider chain (registry order, filtered by circuit state, reordered by the application's routing strategy) and attempt each in order. See Model Registry & Routing and Failover & Circuit Breaking.
8. Response — the client gets its response (or a final error) as soon as dispatch resolves.
9. Background logging — usage accounting and the request log row are written after the response is already on its way to the client, wrapped so a logging failure can never surface as a client-visible error.

Streaming is different after the first chunk#

For stream: true requests, failover can only happen before the first chunk reaches the client. The gateway pulls exactly one chunk from a candidate before committing to it: if the provider fails to produce even one chunk, that counts as a retryable failure and the next candidate is tried. Once a chunk has been sent to the client, the connection is committed — a failure from that point just ends the stream, since content already delivered can't be un-sent or silently replaced by a different provider's answer.

Everything downstream of dispatch is fire-and-forget #

Recording usage (recordUsage) and the request log entry (recordRequestLog) both happen in void (async () => { ... })() blocks wrapped in their own try/catch. This is a deliberate reliability property: a Postgres hiccup while writing observability data must never turn into a 500 for a request that otherwise succeeded.