LLM inference explained for app developers: prefill, decode and the KV cache

I build AI features for a living, mostly through APIs: a seven-agent coaching platform, an MCP server over a production database, an LLM that drafts clinical notes. On Myaigi, the multi-agent platform, the total wait for an answer is several model calls added together, so we stream every answer to the React Native app. Users judge speed by whether they see progress.

That made me want to understand what happens inside one of those calls. These are my notes from working through the “Start here” path of Wafer’s GPU performance engineering reading list, written for developers who call models rather than people who write CUDA. I’m an application engineer, not a kernel engineer, so every number below links to the source I took it from.

One request, two phases

When you send a prompt, the model doesn’t read it word by word. During prefill it pushes every prompt token through every layer at once, as large matrix multiplications. The GPU is doing a lot of arithmetic for each byte it loads, which is exactly what GPUs are good at. Prefill produces two things: the first output token, and the KV cache (more on that below).

Decode is different. The model generates one token, appends it, and runs again for the next one. Each step does a small amount of math, but it has to read the model’s weights from GPU memory to do it. A 7-billion-parameter model stored in 16-bit precision is about 14 GB of weights. So producing a single token means streaming roughly 14 GB through the chip. Then it does it again for the next token.

The clearest walkthrough of this I found is the inference chapter of How to Scale Your Model, which follows one request from prefill through decode.

Why decode is memory-bound: the roofline in one paragraph

The roofline model asks one question: how many operations do you do per byte you move? That ratio is called arithmetic intensity. Below a certain ratio the chip waits on memory; above it, the chip waits on its own arithmetic.

Kipply’s Transformer Inference Arithmetic works this through for an A100: about 312 teraFLOPs of 16-bit compute against about 1.5 TB/s of memory bandwidth, a ratio of roughly 208. You need around 200 operations per byte loaded to keep the math units busy. Decode for a single request does about two operations per weight while loading two bytes per weight. That’s about one operation per byte, nowhere near 200. The GPU spends most of decode waiting for memory.

This one fact explains most of the tricks serving engines use.

The KV cache, and why it eats memory

Attention needs every earlier token’s keys and values. Recomputing them at every decode step would be wasteful, so the engine keeps them in GPU memory: the KV cache. Its size per token is:

bytes per token = 2 (keys and values) × layers × KV heads × head dimension × bytes per value

For Llama 2 7B (32 layers, 32 heads of dimension 128, 16-bit values) that’s 2 × 32 × 32 × 128 × 2 = 524,288 bytes, about 0.5 MB per token. A 4,096-token conversation holds around 2 GB of KV cache on top of the weights, for one user.

That’s why newer models use grouped-query attention, where several query heads share one set of keys and values (Llama 2 70B uses 8 KV heads for its 64 query heads), and why DeepSeek-V2 compresses the cache further. It’s also why vLLM’s PagedAttention manages KV memory in fixed-size pages, the way an operating system manages RAM, so memory isn’t wasted on space a conversation might never use.

How serving engines win speed back

Batching

If decode is waiting on memory anyway, serve many users per trip. The weights are read once per step and used for every request in the batch, which raises arithmetic intensity. Orca introduced iteration-level scheduling, now called continuous batching: new requests join the batch between decode steps instead of waiting for the whole batch to finish.

Quantization

Storing weights in 8 or 4 bits instead of 16 means fewer bytes per decode step, and decode is memory-bound, so it gets faster almost directly. GPTQ and AWQ are the standard weight-only methods.

Speculative decoding

A small draft model guesses several tokens ahead, and the large model checks all of them in a single pass. When the guesses are right you get several tokens for the price of one big-model step, and the original paper shows the output distribution stays exactly the same.

Chunked prefill

One user’s 50-page prompt shouldn’t freeze everyone else’s streaming. Sarathi-Serve splits long prefills into chunks and interleaves them with other users’ decode steps.

What this changes in your app

You can’t tune a provider’s GPUs. You can change what you send them.

1. Stream every answerTime to first token is what users feel. Total time barely matters if text starts appearing in under a second. Here’s how I stream LLM responses into React Native.
2. Measure TTFT and tokens per second separatelyA slow first token points at long prompts or queueing. Slow streaming points at model size or a busy provider. The Etalon paper is a good reference for these serving metrics.
3. Keep your prompt prefix stablePut the system prompt, tools and fixed instructions first and the user’s message last. Providers with prompt caching, and engines with prefix reuse like SGLang, can then skip prefill for the part that repeats.
4. Ask for fewer output tokensEach output token is one decode step, and decode is the slow, sequential part. A structured JSON answer with the three fields you need beats a friendly paragraph.
5. Use small models for small jobsOn Myaigi a fast, cheap model classifies intent and routes the request; stronger models only run where quality matters.
6. Treat long context as a costA bigger prompt means a longer prefill and a bigger KV cache for the whole conversation. Retrieve the three relevant documents instead of pasting in thirty.

Where to go next

If you want to go deeper than app level, the reading order that worked for me:

• How to Scale Your Model: Inference, for the whole request lifecycle.
• Transformer Inference Arithmetic, for the back-of-the-envelope math.
• GPU Mode lectures, when you want to see kernels.
• The full Wafer reading list, which goes from a single request down to kernels and distributed serving.