Calling a model is one line. Serving one is not.
I wanted to properly understand what happens between "here's a model" and "a thousand people are hitting it at once." So I built it up one stage at a time, and measured each stage against the last: a plain script, then a naive endpoint, then a threadpool, then dynamic batching, then caching and backpressure, then metrics.
Every earlier stage is still in the repo, so the comparisons below stay reproducible.
- Validate + check the cache. Seen this exact text before? Return instantly. No queue, no model.
- Join a bounded queue. If it's full, you get a clean
503right away instead of waiting forever. - Get batched. The batch loop grabs up to
MAX_BATCH_SIZErequests or waitsMAX_WAIT_MS, whichever comes first. - One model call for the whole batch, off the event loop, so the server stays responsive.
- Resolve each future, cache it, return it. Latency, queue depth, batch size, and cache hits all flow out to
/metrics.
The numbers go up as the batch grows.
Batch size 1 performs the same as naive, and that's expected: a "batch" of one has nothing to amortize. From 4 → 8 → 16, the fixed cost of each model call gets spread over more requests. 2.7× the throughput and 4× lower p99 than the naive server. Latency drops too, because at this load the server is bottlenecked on the queue, so clearing it faster helps everyone.
Under overload (queue capped at 5, 300 requests at concurrency 200), 276 requests got a clean 503 and the p99 stayed bounded at about 1.2 s. No crash, no unbounded pile-up.
Sandbox run: synthetic same-architecture model, 150 requests at concurrency 24, 4 vCPUs. Raw JSON is in the repo.
PyTorch + asyncio = SIGBUS (on my Mac).
Calling the model from inside FastAPI's event loop crashed every single time with a SIGBUS. Even one forward pass. The same code ran fine as a plain script. I ruled out, one by one:
The crash followed the execution context (event loop or background thread), not the model or the version. The fix was a Linux container, which was on the roadmap anyway. Later, the same code ran clean in a normal Linux environment, which confirmed it was this machine's native threading stack.