Lessons learned after six months of routing 40M tokens a day

The first time a streaming LLM gateway fell over in front of a customer, I had a dashboard that told me nothing. That is the story below.

The short version

p95 time-to-first-token dropped from 1.9s to 640ms after we stopped buffering

What we tried

Move	What happened
Bigger context window	Cost up 3x, accuracy flat
Prompt rewrite on its own	+4% on the golden set, then regressed two weeks later
Swapping Fastify for a boring alternative	Same quality, fewer moving parts

What broke

per-key token buckets are useless if you count tokens after the stream finishes

Behind almost every incident was a default we inherited and never questioned. A token limit too small. A timeout copied from an HTTP tutorial. A retry loop with no ceiling.

By the numbers

• p50 looked fine the entire time; p99 is where users actually lived
• 60% of "the model is wrong" tickets were retrieval returning the wrong document
• Rolling back a prompt was faster than fixing one — every single time

What I would do differently

1. Instrument first: inputs, outputs, latency, token counts, on day one
2. Build a 200-example golden set before the second feature, not after the fifth
3. Make every external call idempotent and deadline-aware
4. Read my own traces for an hour a week, on the calendar

The uncomfortable lesson

Most of what we shipped was ordinary backend engineering wearing an AI hat. None of this was clever. It was just the work nobody wanted to do first.