Blog
Do Quantization and Sparsity make LLM Inference Faster ?
Smaller weights create an opportunity for faster inference, but the kernel, matrix size, and batch size decide whether that opportunity becomes real speed.
Measuring Energy Use in LLM Inference
NF4 drew less board power but used more gross energy per token. Subtracting the GPU idle floor reversed the comparison, while batching produced the largest win.
How much KV cache can you prune before quality degrades?
A recent-window cache removed one-third of the context with almost no perplexity change, while bounded caches kept long-context memory and decode cost flat.
Measuring LLM Serving Performance Under Load
First-token latency failed before decode latency, and fixed bursts showed why average request rate is not enough for capacity planning.