Lessons learned: 3,000 AI review comments, ranked by usefulness

Eighteen months of AI code review, described the way I would want it described to me: numbers, failures, then opinions.

The short version

review noise dropped 70% once the bot stopped commenting on style

What we tried

Move	What happened
Bigger context window	Cost up 3x, accuracy flat
Prompt rewrite on its own	+4% on the golden set, then regressed two weeks later
Swapping GitHub Actions for a boring alternative	Same quality, fewer moving parts

What broke

whole-file prompts invite the model to rewrite your architecture

Behind almost every incident was a default we inherited and never questioned. A token limit too small. A timeout copied from an HTTP tutorial. A retry loop with no ceiling.

By the numbers

• p50 looked fine the entire time; p99 is where users actually lived
• 60% of "the model is wrong" tickets were retrieval returning the wrong document
• Rolling back a prompt was faster than fixing one — every single time

What I would do differently

1. Instrument first: inputs, outputs, latency, token counts, on day one
2. Build a 200-example golden set before the second feature, not after the fifth
3. Make every external call idempotent and deadline-aware
4. Read my own traces for an hour a week, on the calendar

The uncomfortable lesson

Most of what we shipped was ordinary backend engineering wearing an AI hat. Copy it, change the limits, put the verification in CI, move on.