LLM Inference from Scratch
A series in 3 parts, best read in order.
- How LLMs Generate Text, and What the KV Cache Actually Fixes
Generation is a sequential loop, the naive version is quadratic, and the KV cache should make it 260x faster. Measured on GPT-2 built from scratch, it makes it 6x faster. Here is why.
- Why LLM Decoding Is Slow, and Why Batching Is (Almost) Free
A 100-token answer from GPT-2 needs 60 ms of math and takes 2 seconds. Following the bottleneck from a faster chip to the roofline, batching, the KV cache, GQA and goodput, measured on an inference engine built from scratch.
- Batching LLM Requests: Static, Dynamic and Continuous
Static batching at output-length CV of about 1 keeps only 33% of its decode slots useful, and throughput falls from 168 to 61 tok/s. Each fix, from timeouts to continuous batching to chunked prefill, creates the next problem, measured on a GPT-2 engine built from scratch.