Lessons learned: the five retrieval bugs that reached production

What hybrid retrieval for RAG cost us, what it bought us, and the four things I would change tomorrow.

The short version

recall@10 went 0.71 → 0.89 the day we stopped trusting pure vector search

What we tried

Move	What happened
Bigger context window	Cost up 3x, accuracy flat
Prompt rewrite on its own	+4% on the golden set, then regressed two weeks later
Swapping pgvector for a boring alternative	Same quality, fewer moving parts

What broke

dense retrieval silently ignores exact identifiers like account numbers

Behind almost every incident was a default we inherited and never questioned. A token limit too small. A timeout copied from an HTTP tutorial. A retry loop with no ceiling.

By the numbers

• p50 looked fine the entire time; p99 is where users actually lived
• 60% of "the model is wrong" tickets were retrieval returning the wrong document
• Rolling back a prompt was faster than fixing one — every single time

What I would do differently

1. Instrument first: inputs, outputs, latency, token counts, on day one
2. Build a 200-example golden set before the second feature, not after the fifth
3. Make every external call idempotent and deadline-aware
4. Read my own traces for an hour a week, on the calendar

The uncomfortable lesson

Most of what we shipped was ordinary backend engineering wearing an AI hat. That is it. No framework required.