Model Registry & Routing
The registry
Models are declared once, centrally, in apps/gateway/server/registry/models.config.ts:
Authorization: Bearer lattice_sk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxidstring, requiredmodel field). provider'openai' | 'anthropic' | 'google' | 'ollama' | 'vllm'providerModelstring, requiredclaude-sonnet-4-5). fallbacksstring[], optionalids to try if this one's provider fails. Flat, not recursive — a fallback's own fallbacks are not chained in. aliasesstring[], optionalcompany/smart). An alias can never resolve to a different entry, and can't itself be used as a fallback target. pricing{ inputPerMillionUsd, outputPerMillionUsd }, optionalcost or balanced sort and its request logs show no estimated_cost_usd. Boot-time validation
The registry is validated once, at module load — before the server accepts any request. Any of the following crashes startup with a model_registry_invalid error instead of surfacing as a runtime failure on some unlucky request:
- Duplicate
idacross entries. - An
aliascolliding with an existingid, or duplicated across entries. - A model listing its own
idin its ownfallbacks(self-reference). - A
fallbacksentry that doesn't match any knownid(aliases are not valid fallback targets). - The same fallback listed twice in one entry's
fallbacks. - A
providerwith no configured credentials (see Providers for what "configured" means per provider).
Resolving a request
- Lookup:
lookupModel(registry, request.model)matches against every entry'sidfirst, then every entry'saliases. - Candidate order:
[primary, ...primary.fallbacks], flattened — the order declared in the config, unless overridden by strategy. - Circuit filter: candidates whose provider is currently
openare dropped. If that would leave zero candidates, the original, unfiltered order is used instead — a stuck circuit should never be worse than no circuit breaker at all. See Failover & Circuit Breaking. - Strategy reorder: the remaining candidates are sorted by the application's
routingStrategy. - Attempt loop: candidates are tried in the resulting order until one succeeds or all are exhausted. See Request Lifecycle for retry semantics.
Routing strategies
An application can pin a routingStrategy (set through the Management API):
| Strategy | Candidate order |
|---|---|
| unset (default) | The registry's declared [primary, ...fallbacks] order. |
cost | Ascending blended $/M input+output price. |
latency | Ascending rolling-average latency per model. |
balanced | Cost and latency, each min-max normalized to [0, 1], then averaged evenly (0.5 × cost + 0.5 × latency). |
A candidate with no price or no latency sample yet is still tried — it just sorts after every candidate that has a real score, rather than being dropped. Ties and fully-unscored groups preserve their original relative order (a stable sort).
Latency tracking
Every successful provider call records its latency against providerLatencyState, keyed by the registry model id, as a rolling average. That average is what latency and balanced strategies sort on. It's periodically synced across gateway instances the same way circuit breaker state is — see the next page.
Cost estimation
estimateCost(entry, usage) returns undefined unless the entry has pricingand the response reported both promptTokens and completionTokens:
type ProviderAdapter = {
id: ProviderId;
createChatCompletion(
request: ChatCompletionRequest,
ctx: ProviderRequestContext,
): Promise<ChatCompletionResponse>;
streamChatCompletion(
request: ChatCompletionRequest,
ctx: ProviderRequestContext,
): AsyncGenerator<ChatCompletionChunk, void, void>;
};
type ProviderRequestContext = {
apiKey?: string;
providerModel: string;
signal: AbortSignal;
}; This value is what populates estimated_cost_usd on request logs and the usage summary endpoints.