Marcus Reid
@marcusOpen-source maintainer · local inference
Running useful models on hardware people already own. Quantisation, GGUF, and patience.
on Tutorial: a diff-scoped AI reviewer that doesn't spam your PR · 5d ago
Any advice on making the error taxonomy legible to support? Ours still asks what 'degraded' means.
on Tutorial: hybrid search with pgvector and reciprocal rank fusion · 5d ago
We hit the same problem at 3x and fixed it by sharding the queue rather than the workers.
on Tutorial: hybrid search with pgvector and reciprocal rank fusion · 5d ago
Would be interested to hear how that held up after a few months.
on Tutorial: hybrid search with pgvector and reciprocal rank fusion · 5d ago
The tuning-knobs table is more useful than most vendor documentation.
on Tutorial: hybrid search with pgvector and reciprocal rank fusion · 6d ago
We road-tested something similar; the honest answer is it's 20% design and 80% discipline.
on Tutorial: hybrid search with pgvector and reciprocal rank fusion · 6d ago
This is what I wish our vendor's onboarding looked like.
on Tutorial: build a streaming LLM gateway with per-tenant rate limits · 8d ago
Do you version prompts with the code, or separately? We're still arguing about it.
on Lessons learned: 40k requests a day on a single 24GB card · 13d ago
Careful with that; it bit us when the upstream started returning 200s with empty bodies.
on Architecture notes: evals, tracing and logs are one system · 16d ago
One gap: nothing here about data residency. That's what killed our multi-provider plan. (82)
on Architecture notes: tool schemas as a product surface · 17d ago
Any advice on making the error taxonomy legible to support? Ours still asks what 'degraded' means. (3)
on Tutorial: a golden-set eval suite that blocks bad prompt merges · 23d ago
We run something similar at roughly a third of this volume and hit the same wall around 60 concurrent streams. (6)
on Architecture notes: one GPU node pool, three very different workloads · 24d ago
This deserves its own post.