DOCS / PROVIDERS / ADDING A PROVIDER

Adding a Provider

Two paths, depending on what the upstream API looks like.

It already speaks OpenAI's `/v1/chat/completions` shape

This covers most self-hosted inference servers (Ollama, vLLM, and many others built on the same convention). No new adapter code is needed — configure it exactly like Ollama & vLLM: point a baseUrl env var at it and reference it from the registry. If it needs a fixed, non-configurable base URL (rather than reading one from the environment), add a one-line wrapper around createOpenAiCompatibleAdapter following providers/ollama/adapter.ts as a template.

It needs bespoke request/response mapping

Follow the shape of providers/anthropic/ or providers/google/:

Define the adapter's request/response/stream mapping

Create server/providers/<name>/{request,response,stream}.ts:

request.ts — maps ChatCompletionRequest (Lattice's shape) into whatever body the upstream API expects. Handle anything the upstream API doesn't support natively the way Anthropic's adapter handles system messages: transform, don't drop silently.
response.ts — maps the upstream's non-streaming response back into ChatCompletionResponse, including a finish_reason normalized to 'stop' | 'length' | 'content_filter'.
stream.ts — maps the upstream's SSE stream into AsyncGenerator<ChatCompletionChunk>. Reuse parseDataOnlySseStream or parseNamedEventSseStream from providers/sse.ts depending on whether the upstream sends self-contained JSON payloads or named events. If the upstream's stream reports usage/finish incrementally rather than as one final self-contained message, model it as a stateful reducer the way anthropic/stream.ts does.

Implement the adapter

Create server/providers/<name>/adapter.ts exporting a ProviderAdapter:

adapter.ts

Authorization: Bearer lattice_sk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Always forward ctx.signal to fetch — dispatch relies on aborting the in-flight request when a non-streaming attempt times out or a streaming client disconnects. Always call assertProviderOk before parsing a response body, so upstream errors surface as ProviderRequestError and participate correctly in circuit breaker retry logic.

Add the provider id and credential resolution

Extend the ProviderId union in providers/types.ts, and add a branch in registry/credentials.ts's buildProviderCredentials describing what env var(s) make this provider "configured" — an API key for a cloud-style provider, a base URL for a self-hosted one.

Register the adapter and reference it from the registry

Add the adapter to the adapters map in routing/deps.ts, then add a model entry in registry/models.config.ts with provider: 'my-provider'.

Write tests

Mirror the existing provider test layout under tests/apps/gateway/server/providers/<name>/: one test file per mapping concern (adapter, request, response, stream), following the pattern in tests/apps/gateway/server/providers/anthropic/*.

Checklist

ProviderAdapterimplemented

id, createChatCompletion, streamChatCompletion — all forwarding ctx.signal.

Errorsnormalized

Non-2xx responses raised via assertProviderOk so failures are retryable/circuit-tracked correctly.

finish_reasonnormalized

Every upstream stop/finish value maps to 'stop' | 'length' | 'content_filter'.

Credentialsresolved

credentials.ts knows what env var(s) make this provider count as configured, so registry boot validation can enforce it.

Testsadded

Adapter, request, response, and (if streaming) stream mapping each covered.

← Ollama & vLLM Authentication →