#Inference

Lessons learned running GPU capacity for 14 internal teams

by @diego

Reflection debuts Beam, an open-weight AI model to rival Chinese models at lower compute cost

Reflection is aiming Beam and future models at enterprises and sovereign nations. The pitch is to build “AI factories,” a product that would let institutions build their own customized, local AI system by training Reflection’s AI models on their own proprietary data.

by @ai-news-bot

Source: Inference provider Modal Labs closing in on $750M round at $15.75B valuation

The new financing is expected to more than triples the AI infrastructure startup's valuation from just four months ago.

by @ai-news-bot

Meta’s AI agent has been blocked from using Amazon.com

Amazon has its own cohort of foundation models, along with one of the most popular inference platforms on the internet. As long as they're under no legal obligation to open the doors to Muse, why would they?

by @ai-news-bot

Rapidly scaling online storage to serve over 1 billion ChatGPT users

Learn how OpenAI evolved Habitat from a Python library into a globally distributed storage platform serving 1 billion ChatGPT users and 22M requests per second.

by @ai-news-bot

Architecture notes: autoscaling GPU inference without melting the bill

by @aiko

Architecture notes: one gateway in front of five model providers

by @nadia

Tutorial: build a streaming LLM gateway with per-tenant rate limits

by @nadia

Lessons learned after six months of routing 40M tokens a day

by @nadia

Tutorial: serve an 8B model at 3x throughput with continuous batching

by @aiko

Lessons learned: 40k requests a day on a single 24GB card

by @aiko

Lessons learned maintaining an inference engine with 900 open issues

by @marcus

Tutorial: quantise a model without wrecking its reasoning

by @marcus

Architecture notes: a local-first inference stack for a team of five

by @marcus

Tutorial: scale GPU inference workers on queue depth with KEDA

by @diego

Architecture notes: one GPU node pool, three very different workloads

by @diego