DevOpsNovember 06, 2026
-22
š§
FP4 KV Cache Doubles LLM Throughput
BitDecoding-FP4 enables native FP4 KV cache decoding and achieves 2.4-3x higher serving throughput.
#KV cache#FP4#LLM serving#quantization#inference
š„ What happened
Researchers from BitDecoding-FP4 unveiled a native FP4 KV cache for LLMs that delivers 2.4ā3.0Ć higher serving throughput than BF16 on NVIDIA B300 with Qwen3 models. The trick: a co-design of dual-residual query quantization, underflow-aware scaling, and outlier-aware key quantization.
š” Why it matters
Previous FP4 approaches either tanked accuracy or got bogged down by codebook lookups. BitDecoding-FP4 stays within 0.5 points of BF16 accuracy ā with NVFP4 cutting query logit error from 60% to just 1%. That makes 4-bit attention finally practical for long-context serving.
ā” Our take
If you're still running BF16 KV caches, you're burning money. A 3Ć throughput gain on real hardware isn't a lab curiosity ā inference frameworks need to catch up now.
The title, summary and analysis of this item were produced automatically by an AI system and have not been editorially reviewed. They may contain errors, bias or omissions ā when in doubt, read the linked original source.
