Lessons learned maintaining an inference engine with 900 open issues

What local inference on consumer GPUs cost us, what it bought us, and the four things I would change tomorrow.

The short version

Q4_K_M cut memory from 16GB to 5.2GB with a 1.8% eval drop

What we tried

Move	What happened
Bigger context window	Cost up 3x, accuracy flat
Prompt rewrite on its own	+4% on the golden set, then regressed two weeks later
Swapping llama.cpp for a boring alternative	Same quality, fewer moving parts

What broke

quantising the wrong tensors hurts more than dropping a bit width

Behind almost every incident was a default we inherited and never questioned. A token limit too small. A timeout copied from an HTTP tutorial. A retry loop with no ceiling.

By the numbers

• p50 looked fine the entire time; p99 is where users actually lived
• 60% of "the model is wrong" tickets were retrieval returning the wrong document
• Rolling back a prompt was faster than fixing one — every single time

What I would do differently

1. Instrument first: inputs, outputs, latency, token counts, on day one
2. Build a 200-example golden set before the second feature, not after the fifth
3. Make every external call idempotent and deadline-aware
4. Read my own traces for an hour a week, on the calendar

The uncomfortable lesson

Most of what we shipped was ordinary backend engineering wearing an AI hat. That is it. No framework required.