Retrospective · numbers first
Lessons learned: 6% of our first LLM outputs were quietly malformed
by @leila — Applied research → production
A retrospective on structured outputs from LLMs — the headline number, what broke, and the four changes I would make tomorrow.
Eighteen months of structured outputs from LLMs, described the way I would want it described to me: numbers, failures, then opinions.
The short version
schema violations dropped from 6% to 0.2% by validating before persisting
What we tried
| Move | What happened |
|---|---|
| Bigger context window | Cost up 3x, accuracy flat |
| Prompt rewrite on its own | +4% on the golden set, then regressed two weeks later |
| Swapping Zod for a boring alternative | Same quality, fewer moving parts |
What broke
asking for JSON in the prompt is not the same as constrained decoding
Behind almost every incident was a default we inherited and never questioned. A token limit too small. A timeout copied from an HTTP tutorial. A retry loop with no ceiling.
By the numbers
- p50 looked fine the entire time; p99 is where users actually lived
- 60% of "the model is wrong" tickets were retrieval returning the wrong document
- Rolling back a prompt was faster than fixing one — every single time
What I would do differently
- Instrument first: inputs, outputs, latency, token counts, on day one
- Build a 200-example golden set before the second feature, not after the fifth
- Make every external call idempotent and deadline-aware
- Read my own traces for an hour a week, on the calendar
The uncomfortable lesson
Most of what we shipped was ordinary backend engineering wearing an AI hat. Copy it, change the limits, put the verification in CI, move on.
Comments (2)
Log in to join the discussion.