Tutorial: build a streaming LLM gateway with per-tenant rate limits

Before you start — about 45 minutes · Fastify · Redis · two provider SDKs

We rebuilt a streaming LLM gateway three times before it held. The version below is the one that survived a bad Tuesday and a worse Friday.

Step 1 — Decide what "working" means

Before touching code, write down three numbers: worst acceptable latency, cost per request, and what should happen when the upstream is unavailable. Everything after this is implementation.

Step 2 — Wire Fastify

The whole flow is one path. No special branches for "large" inputs — express those as different limits, not different code.

// token bucket that reserves budget BEFORE the request hits the provider
async function reserve(key: string, estimate: number) {
  const lua = `
    local used = tonumber(redis.call('GET', KEYS[1]) or '0')
    local limit = tonumber(ARGV[1])
    if used + tonumber(ARGV[2]) > limit then return 0 end
    redis.call('INCRBY', KEYS[1], ARGV[2])
    redis.call('EXPIRE', KEYS[1], 60)
    return 1
  `;
  return (await redis.eval(lua, 1, `quota:${key}`, limit, estimate)) === 1;
}

Why this works

The limit is enforced before the expensive call, not after it. Reserving budget up front means a burst degrades the caller that caused it instead of everyone downstream.

Step 3 — Guard the failure path

per-key token buckets are useless if you count tokens after the stream finishes

Write that guard before you write the feature. It costs an hour and saves a postmortem.

Step 4 — Verify it

curl -N localhost:8080/v1/chat -H 'x-tenant: acme' \
  -d '{"model":"small","stream":true,"messages":[{"role":"user","content":"hi"}]}'

• Lower the limit and confirm the command fails loudly
• Kill the upstream and confirm you get a typed error, not a timeout
• Check the metric you emit per outcome: ok, throttled, upstream_error

Tuning knobs

Knob	Default	What we run	Why
concurrency	8	24	Most of the wait is network, not compute
retry budget	infinite	2, with jitter	Unbounded retries turn an outage into an avalanche
timeout	30s	8s, inherited from the caller	Nobody waits 30s for a chat reply

None of this was clever. It was just the work nobody wanted to do first.