What breaks first when an LLM server is overloaded?
Offline benchmarks hand an inference engine a fixed batch and wait for it to finish. A real server receives requests over time. Some arrive steadily, while others arrive together in bursts. As demand increases, throughput may continue to rise even while users wait too long for useful responses. The question was: what fails first under steady and bursty load, and which vLLM scheduling knobs move the useful-capacity boundary?
Why this matters
Tokens per second does not say whether individual requests meet a latency goal. A server can report higher raw throughput while a growing queue makes most of that work too late to be useful. Capacity therefore needs an explicit service level objective, or SLO, and it needs an arrival model that reflects bursts.
Main finding
Under Poisson arrivals, the server met a 1-second time-to-first-token and 100-millisecond per-output-token SLO through 2 requests per second. It first failed at 3 requests per second because first-token latency crossed the limit before decode latency. Fixed bursts of eight requests missed the same SLO even at only 0.5 requests per second on average.
The vertical boundary is defined by useful requests that meet both latency limits, not by raw token throughput.
What we did
I ran Qwen2.5-1.5B behind a real vLLM OpenAI-compatible server on one RTX 3060 12 GB GPU. An asynchronous client generated 4,608 requests across 96 matched runs. Prompt lengths ranged from 128 to 4,096 tokens, output lengths ranged from 32 to 256 tokens, and offered load ranged from 0.5 to 6 requests per second.
Two arrival processes used the same seeded request lengths:
- Poisson traffic with variable gaps between individual requests
- fixed groups of eight simultaneous requests followed by a quiet interval
The SLO required time to first token, TTFT, at or below 1,000 ms and time per output token, TPOT, at or below 100 ms. Goodput counted only requests meeting both limits. Server metrics tracked queue depth, running requests, KV-cache use, and preemptions.
Four server configurations compared the baseline with a lower
max_num_seqs, half the GPU-memory reservation, and chunked prefill disabled.
Detailed results
| Poisson load | TTFT p95 | TPOT p95 | SLO attainment |
|---|---|---|---|
| 1 request/s | 812 ms | 22.8 ms | 99.0% |
| 2 requests/s | 979 ms | 33.6 ms | 92.7% |
| 3 requests/s | 1,160 ms | 51.5 ms | 89.6% |
| 6 requests/s | 2,142 ms | 179.1 ms | 33.3% |
At 3 requests per second, TPOT was still well below its limit while TTFT had already crossed 1 second. The request waited too long to begin, even though generation proceeded quickly once admitted.
For eight-request bursts, reducing the mean rate only increased the quiet time between groups. It did not make the arriving group smaller. At the lowest tested mean rate, TTFT p95 was already 1.67 seconds and only 45.8% of requests passed.
Limiting concurrent sequences protected decode latency under overload by moving more requests into the waiting queue.
None of the tested knobs moved the 2 requests-per-second SLO capacity. At 6
requests per second, limiting max_num_seqs to 16 improved TPOT p95 from 179
to 71 ms, but doubled TTFT p95 from 2.14 to 4.28 seconds and increased the
maximum queue from 6.5 to 15.5 requests. It changed who waited; it did not add
capacity.
What we learned
First-token latency is the early warning signal for this workload. Raw throughput keeps rising after useful capacity has been exceeded, so it can place the serving knee too late.
Average request rate is also insufficient for bursty traffic. A quiet interval after a burst improves the average but does nothing for requests that arrived together. Admission and batching policies must account for burst size, not only long-run requests per second.
Limitations
The numerical capacity belongs to one GPU, one 1.5B model, the tested length distribution, and the chosen SLO. The study used two seeded workload repeats, not a statistical confidence interval. Burst size was fixed at eight, so the experiment shows that eight is too large but does not find the largest safe burst. Prefix caching was disabled to keep repeated synthetic prompt structure from becoming a hidden advantage.
Future work
The next steps are to sweep burst size, add trace-driven arrivals, test larger models, and evaluate adaptive admission policies that optimize useful goodput instead of raw throughput. Energy measurement under the same request traces would connect serving quality with operating cost.
Technical evidence
Read the complete report, code, raw measurements, and reproduction instructions on GitHub.