Architecture notes: one gateway in front of five model providers

ADR-012 · Status: accepted after two rejected drafts. The argument that won is on this page.

Context

We run a streaming LLM gateway for a product that cannot afford a bad week. This document records the constraints we were given, the shape we chose, and what we knowingly gave up.

Constraints

Constraint	Target	Why it matters
p95 latency	< 2s	Below this, the product feels broken
Failure isolation	one tenant only	A noisy neighbour must not page the on-call
Cost visibility	per request	Finance asks monthly, and they are right
Change safety	revert in minutes	Prompts are code, so they need rollbacks

Decision

Put one narrow boundary in front of the model, keep retries and fallbacks inside it, and emit telemetry from the same request path.

Three boundaries, each with exactly one job:

1. Edge terminates identity and budget. Nothing downstream re-implements auth.
2. Workers own retries, deadlines and fallbacks. Stateless, horizontally scalable.
3. Telemetry is written inline — never reconstructed from logs afterwards.

Structure

client
  │
  ├── edge: auth · quota · tenant routing
  │        │
  │        ├── Fastify
  │        └── Redis
  │
  └── workers → upstream providers
           │
           └── telemetry: traces · evals · cost

Interfaces

The contract between edge and workers is a single schema, versioned with the deploy. Fastify + Redis + two provider SDKs stay interchangeable because that contract never changes.

Consequences

What gets better

• Quotas are explainable to customers: one number, per tenant, observable.
• A provider outage becomes a routing decision instead of an incident.
• Every request has a cost, a latency and an outcome attached to it.

What gets worse

• One more hop of latency, roughly 8ms on our hardware.
• The edge becomes a shared dependency — it needs its own on-call rotation.
• Deploying it requires coordination with every consumer.

Failure modes

Under 3x load the first thing to fail was not the model; it was retries amplifying a partial outage. Every retry now carries jitter and a deadline inherited from the caller.

per-key token buckets are useless if you count tokens after the stream finishes

Alternatives we rejected

Option	Why it lost
Library inside each service	Auth and quota implemented five times, five ways
Provider SDK directly	No shared budget, no shared telemetry
Gateway as a product (vendor)	Cost per token on top of cost per token

Operations

• Dashboards per tenant, not per instance
• One alert that matters: error rate above 1% for five minutes
• A runbook entry for "upstream degraded" a new hire can follow at 3am

None of this was clever. It was just the work nobody wanted to do first.