Does compression actually make LLM inference faster?
Large language models can be compressed by storing weights with fewer bits or by removing values in a structured pattern. That reduces memory traffic, which sounds like it should make inference faster. But the GPU does not execute a storage format directly. It executes a kernel, and that kernel may introduce dequantization, metadata handling, unsupported shapes, or launch overhead. The question was: when do compressed representations turn fewer bytes into less execution time?
Why this matters
Batch-1 LLM decoding repeatedly streams model weights from GPU memory. If weight traffic is the bottleneck, cutting those bytes should help. This is why quantization and sparsity are often described as performance optimizations. That description hides an important condition: the hardware and software stack must provide an efficient kernel for the exact matrix shape and batch size.
Main finding
Compression was not a universal speedup. NF4 was 1.75x faster than dense FP16 at batch 1 but became 2.09x slower at batch 8. INT8 was unsupported at the two smallest batches, then reached a 1.77x speedup at batch 128. The library 2:4 sparse path was slower at every tested batch.
Weighted device time across the 197 matrix multiplications in one Qwen2.5-1.5B decode step. Values above 1 mean faster than dense FP16.
What we did
I extracted the eight matrix shapes used by a real Qwen2.5-1.5B decode step and tested each at batch sizes 1, 8, 32, and 128. The formats were:
- dense FP16 through cuBLAS
- INT8 through PyTorch’s integer matrix multiplication path
- NF4 through bitsandbytes
- 2:4 semi-structured sparse FP16 through cuSPARSELt
Each result was measured on one RTX 3060 12 GB GPU with compute capability 8.6. I first measured the card’s streaming-read ceiling at 341.4 GB/s, then built a bytes-per-output model for each representation. Every timing used CUDA events, discarded warmups, repeated samples, and an L2-cache flush before measurement.
I also wrote a custom CUDA 2:4 sparse GEMV kernel for batch 1. It stores two FP16 values out of each group of four and packs their positions into metadata. This let the experiment distinguish a limitation of compression itself from a limitation of the available library kernel.
Detailed results
| format | batch | weighted time | speedup over FP16 |
|---|---|---|---|
| dense FP16 | 1 | 10.86 ms | 1.00x |
| NF4 | 1 | 6.20 ms | 1.75x |
| library 2:4 | 1 | 31.28 ms | 0.35x |
| custom CUDA 2:4 | 1 | 10.26 ms | 1.06x |
| INT8 | 32 | 8.01 ms | 1.55x |
| INT8 | 128 | 11.24 ms | 1.77x |
The byte model worked best when execution stayed close to one streaming kernel. Dequantization and fixed kernel costs created the largest misses.
The NF4 representation did not change between batches, but its execution path did. At batch 1, bitsandbytes selected a fused 4-bit GEMV. At larger batches it dequantized the full weight matrix before a dense operation, which erased the traffic advantage.
INT8 showed the opposite pattern. The tested PyTorch kernel rejected batch dimensions at or below 16. Once the matrix was large enough to use its supported path, INT8 became the fastest tested representation.
The custom sparse kernel recovered most of the library path’s fixed cost on small projections and produced a 1.45x speedup on the model’s down projection. Across the whole decode step it was only 1.06x faster, showing that storage savings alone do not supply good scheduling, vectorization, or tensor-core use.
What we learned
The useful unit of analysis is not just the format. It is the combination of format, matrix shape, batch size, hardware, and selected kernel. A compressed representation creates an opportunity to move fewer bytes. Whether that opportunity becomes speed depends on the execution path.
This also explains why a benchmark at one batch size cannot establish that a format is generally faster. NF4 won the latency regime while INT8 won the larger-batch throughput regime on the same card.
Limitations
This was a kernel-level execution study on one model, one GPU, and one software stack. It did not measure model quality after quantization or pruning. INT8 activation-quantization cost was excluded, and the custom CUDA kernel supports only batch-1 FP16 2:4 GEMV. Different libraries or newer kernels may move these boundaries.
Future work
The next step is to improve the custom sparse kernel with vectorized loads, multiple outputs per block, and tensor-core paths, then measure the formats inside complete end-to-end generation rather than only the projection kernels.
Technical evidence
Read the complete report, code, raw measurements, and reproduction instructions on GitHub.