Here is a number that surprised me the first time I worked it out. An NVIDIA H100 can do about 989 trillion bf16 operations per second [4]. Generating one token with Llama 3.1 8B takes about 16 billion of them, two for each of its 8 billion weights. On paper, the chip could write around 60,000 tokens a second.

Serve that model to one user and it writes at most about 200. The reason is not arithmetic. To produce each token, the GPU has to read almost all 16 GB of weights out of memory, and at 3.35 TB/s [4] that takes about 5 milliseconds. The math itself takes 16 microseconds. For the rest of each step, the arithmetic units sit idle, waiting for bytes.

Step through it and watch the meter. Each pass reads the weights again, and uses well under 1% of the chip’s arithmetic:

LLM inference is mostly a fight with memory traffic. Almost every trick either moves fewer bytes per token, or gets more tokens out of every byte it moves.

In 2023 I wrote about inference for machine-learning models in general: distillation, quantization, pruning, half precision and ONNX Runtime. Those levers still matter, and two of them come back in §8 and §9. What that post didn’t cover is what makes LLMs different: they write one token at a time, and that loop changes where the time goes.

The post builds up in the order I’d learn it: what one model call does (§1–2), the hardware it runs on (§3), the numbers that follow from both (§4–5), the techniques that bend those numbers (§6–9), and how it all scales and gets packaged into real engines (§10–11).

• 1. What happens when you call an LLM
• 2. Two phases, one model
• 3. Inside the GPU: where the bytes live
• 4. Why decode waits, and how batching helps
• 5. The numbers users feel
• 6. The KV cache: trading memory for speed
  • 6.1 Paging: the cache as virtual memory
  • 6.2 FlashAttention: never write the big matrix
• 7. Speculative decoding: guess cheaply, check in bulk
• 8. Quantization: fewer bytes per weight
• 9. Pruning: fewer weights
• 10. Beyond one GPU: splitting the model
• 11. Inference engines: putting it together
• 12. Where this simple model breaks
• 13. A checklist for a slow deployment
• 14. Where to read next
• Cite this post
• References

1. What happens when you call an LLM

Before the performance story, it helps to see what one call actually computes. Here is the path a prompt takes through a model like Llama 3.1 8B [5][36]:

1. Tokenize. The text is split into tokens, whole words or pieces of words, and each becomes an ID. Llama 3’s vocabulary has 128,256 of them [5].
2. Embed. Each ID picks a row from an embedding table: a vector of 4,096 numbers that the rest of the model works on.
3. Run the transformer blocks. The vectors pass through 32 identical blocks. Each block has two parts:
   • Self-attention. Every token produces three vectors: a query (Q, what it is looking for), a key (K, what it offers) and a value (V, what it passes on). A token’s query is compared with the keys of every token before it, and the resulting weights decide how much of each value to mix in. This is the only place tokens exchange information.
   • MLP. A feed-forward network then transforms each token’s vector on its own.
4. Predict. The final vector is multiplied by an output head into one score per vocabulary entry. Softmax turns those into probabilities, and a sampler picks the next token. Two knobs shape that choice: temperature sharpens or flattens the distribution, and top-p discards the unlikely tail. Try both at stage 6 below.
5. Repeat. The new token is appended, and the model runs again. This loop is autoregressive generation.

Step through it. The bar underneath shows where the model’s weights live, and which part each stage uses:

Two details from this figure come back again and again. First, most of the weights sit in the MLPs (70% in Llama 3.1 8B, more in the bigger models), and every one of them is used for every token. Second, a token’s keys and values are needed again by every later token, but its query is not. That asymmetry is why servers keep a KV cache and never a Q cache. If you want to see the same pipeline in 3D, Brendan Bycroft’s LLM visualization is wonderful [35].

2. Two phases, one model

Every request runs the same weights in two very different ways [1].

In prefill, the model reads your whole prompt in a single forward pass, with all tokens processed in parallel. Along the way, every layer computes the keys and values from §1 for every prompt token. The model stores them in the KV cache, then picks the first token of the answer.

Then decode takes over. It’s a loop: feed in the token just produced, run the model, pick the next one, repeat until done. Each pass handles one new token per conversation, reuses the cached keys and values for everything before it, and adds one more entry to the cache.

Figure 1. The two phases. Prefill uses the weights once for many tokens; decode reads them again for every single token. Diagram by the author, after [1].

That difference decides which hardware limit you hit. A matrix multiply over a long prompt reuses each weight hundreds of times once it’s loaded, so prefill is limited by how fast the chip can multiply. Decode uses each weight once per token, so it is limited by how fast memory can deliver the weights [1][2]. To see why those are two different limits, we need to look inside the GPU.

3. Inside the GPU: where the bytes live

A GPU is many small processors sharing one large memory. An H100 has 132 streaming multiprocessors (SMs). Each SM has its own tensor cores, which do the matrix math, and a small amount of very fast on-chip memory: a 256 KB register file and 256 KB of L1 cache and shared memory, the SRAM that kernels manage by hand. All SMs share a 50 MB L2 cache. The model itself lives off the chip, in 80 GB of HBM (high-bandwidth memory) [30][4].

Figure 2. The memory hierarchy of an H100. The closer memory is to the tensor cores, the faster and smaller it is. Diagram by the author from NVIDIA's specifications [30][4].

The gap between the levels is large. The FlashAttention authors estimate that an A100’s on-chip SRAM delivers around 19 TB/s, against 1.5–2.0 TB/s for its HBM [27]. Every byte a kernel needs has to travel from HBM into an SM before a tensor core can touch it.

Horace He’s Making Deep Learning Go Brrrr splits the time any GPU program takes into three parts [26]:

• Compute: time spent doing math on the tensor cores.
• Memory bandwidth: time spent moving data between HBM and the SMs.
• Overhead: everything else, such as Python, framework dispatch and kernel launches.

Which one dominates depends on arithmetic intensity: how many operations a kernel does for each byte it moves. Divide the chip’s peak FLOPS by its bandwidth and you get the ridge point, the intensity a kernel needs to keep the tensor cores busy. For an H100 that’s 989 ÷ 3.35 ≈ 295 operations per byte [4][31]. Below the ridge, a kernel is memory-bound; above it, compute-bound. Here are the main operations in an LLM, placed against that line:

This one chart explains most of the post. Prefill sits far to the right, so it is compute-bound. Decode with one user sits at about 1, deep in memory-bound territory. Batching drags decode’s weight matmuls to the right, toward the ridge (§4). Attention over the KV cache doesn’t move with batch size at all, because every conversation has its own cache to read; that’s why long contexts hurt (§6). And the elementwise operations, norms and activations, are always memory-bound. That’s why engines fuse them into neighboring kernels, so their inputs never make the round trip to HBM [26][31].

The third bucket, overhead, matters most when the GPU work is small: a small model, a small batch, or decode. Launching the hundreds of kernels in one forward pass costs real time, which is why vLLM records each decode step as a CUDA graph and replays it in one launch [2].

4. Why decode waits, and how batching helps

Put numbers on the decode dot from §3. With one user, each bf16 weight is 2 bytes and is used for 2 operations (a multiply and an add). That’s about 1 operation per byte, against the 295 an H100 needs to keep its tensor cores busy [4]. That ratio is the whole problem in one line.

The fix is to process many conversations in the same pass. Read a weight once, and use it for 32 users instead of one: now you get 32 operations per byte for almost the same time per step. This is batching, and it works until the batch is big enough that arithmetic, not memory, sets the pace. Aleksa Gordić’s tour of vLLM describes this crossover with a simple roofline [2]:

Drag the batch size and watch the step time stay flat, then bend upward at the knee:

This is the trade-off every serving system tunes. Below the knee, adding users is nearly free. Above it, every extra user slows everyone down. Engines keep the batch full with continuous batching. Orca introduced the idea as iteration-level scheduling: the scheduler decides after every step, so finished requests leave and waiting ones join, instead of the whole batch waiting for its slowest member [7][2]. Compare the two schedulers, then add a request with a very long prompt and turn on chunked prefill:

The toy numbers make the trade-offs plain. Continuous batching keeps the slots busy and gets first tokens out sooner. A long prompt still freezes everyone who is mid-answer for one long step, and chunked prefill spreads that cost over several shorter steps (§5).

Batching has a side effect most people never notice: it can change your answer. Many GPU kernels pick a different way to add up numbers depending on the batch size, and floating-point addition isn’t exactly associative. So the same prompt, at temperature 0, can come out differently depending on how busy the server is. Thinking Machines sent one prompt to a large model 1,000 times and got 80 distinct completions. With kernels rewritten to be batch-invariant, all 1,000 matched, at about 1.6× the run time in their lightly optimized version [13].

5. The numbers users feel

Users don’t see FLOPS. They see two delays [1][2]:

• Time to first token (TTFT): from pressing Enter to the first word appearing. This is mostly prefill, so long prompts hurt it.
• Time per output token (TPOT): the average gap between tokens as the answer streams. The gap between any two tokens is called the inter-token latency (ITL). This is decode.

End-to-end latency is about TTFT + output tokens × TPOT. On the server side, throughput counts tokens per second across all users, and goodput counts only tokens from requests that met their latency targets [2]. Try the three presets:

Summarizing a long document is a TTFT problem; writing an essay is a TPOT problem. Notice also the attention share in the summary preset. Prefill cost grows with the square of the prompt length, because every token attends to every token before it, so very long prompts get expensive faster than you’d expect.

Production engines protect TTFT with three ideas:

• Chunked prefill splits a long prompt across several steps, so it can’t freeze everyone else’s decoding while it runs; you saw it in the §4 timeline [2].
• Prefix caching reuses the cached keys and values for a prompt prefix the server has seen before, such as a shared system prompt, and skips that part of prefill entirely [2]. API providers sell the same idea as prompt caching [37]. §6 shows how it works.
• Disaggregated prefill and decode runs the two phases on separate pools of GPUs and ships the KV cache between them, so a burst of long prompts can’t stall steady decoding [16][2].

Here is what this looks like at the top end. A team serving GLM-5.2, a 744-billion-parameter mixture-of-experts model, in 4-bit on 24 B300 GPUs with disaggregated prefill and decode, brought mean TPOT from about 40 ms to 17 ms. The largest single gain, about 40 ms to 22 ms, came from a fix in how speculative decoding (§7) padded its work on the decode side [17]. At that scale, the wins are often in the details of the engine.

6. The KV cache: trading memory for speed

Without a cache, every decode step would recompute the keys and values for the whole conversation so far. That’s quadratic work, and in the opening figure it shows as a counter that runs away. The cache stores them once, so each step computes only the new token’s [1][14]. When Hugging Face added a KV cache to their small nanoVLM model, generation got about 38% faster [14].

The price is memory, and it grows with every token in every conversation:

Why only 8 KV heads when the model has 32 attention heads? That’s grouped-query attention: several query heads share one key/value head, which cuts the cache by 4× here. Its authors report quality close to full multi-head attention at nearly the speed of a single shared head [15]. It’s one reason modern models can afford long contexts at all.

The cache also has to be read on every decode step, so long contexts slow decoding down, not just fill memory. Play with the budget:

6.1 Paging: the cache as virtual memory

Early servers reserved one contiguous slab of cache per request, sized for the longest answer it might produce, and much of that memory was lost to fragmentation and duplicate copies [6]. vLLM’s PagedAttention borrowed the idea of pages from operating systems: split the cache into small fixed blocks (16 tokens by default), give each request a block table that maps its tokens to blocks, and hand out blocks only as tokens arrive. The PagedAttention paper reports near-zero wasted cache memory and 2–4× higher throughput than earlier systems [6][2].

Pages also make sharing easy. vLLM hashes each full block, chained with the hash of the block before it, so two requests that start with the same tokens get the same hashes and can point at the same physical blocks. That’s prefix caching [2]. Step through it:

Turn prefix caching off and replay: the system prompt is stored three times instead of once, and request D pays for the full prefill again.

6.2 FlashAttention: never write the big matrix

The cache solves decode’s recomputation. Prefill has a different attention problem. For a prompt of N tokens, each attention head scores every query against every key: an N × N matrix. A straightforward kernel writes that matrix to HBM, reads it back to apply the softmax, and reads it again to mix the values. At N = 32,768, one head’s matrix in one layer is about 2 GB, and it’s pure memory traffic.

FlashAttention reorganizes the same computation around the memory hierarchy from §3 [27]. It loads a block of queries into SRAM, streams blocks of keys and values past it, and keeps a running softmax as it goes. Each tile of scores is used and thrown away on chip, so the full matrix never exists in HBM. The result is exact attention, not an approximation, with far fewer HBM reads and writes; the paper reports a 3× training speedup on GPT-2 at 1K tokens [27].

You rarely turn this on yourself: vLLM, for example, sets FlashAttention as its attention backend when it starts up [2]. What it teaches carries further: the fastest kernel is often the one that moves the fewest bytes, not the one that does the fewest operations.

7. Speculative decoding: guess cheaply, check in bulk

Back to the opening figure: each decode pass reads about 15 GB to produce one token, and the arithmetic units are nearly idle. Speculative decoding spends that idle arithmetic on checking guesses [8][9].

1. Draft. Something cheap proposes the next k tokens.
2. Verify. The big model runs one forward pass over all k guesses at once. Because decode is memory-bound, that costs about the same as producing a single token, and it yields the big model’s own prediction at every position.
3. Accept. Keep guesses from left to right until the first one the big model disagrees with, and use the big model’s token there. If all k pass, you get one extra token for free.

The acceptance rule is designed so the output follows exactly the same distribution as the big model on its own; the original paper reports 2–3× faster T5-XXL with identical outputs [8][9]. The speedup costs no quality.

The drafter can be a small model of the same family, an n-gram lookup that copies text the conversation already contains, a small network that predicts the big model’s next internal feature vector instead of the next token (EAGLE [10]), extra prediction heads on the big model (Medusa [11]), or multi-token-prediction layers that some models ship with [17]. Try it:

If each guess is right with probability α, independently, a round yields on average (1 − α^k+1) ÷ (1 − α) tokens [8]. Long drafts only pay off when α is high, because one early miss throws away everything after it. Real drafters aren’t independent from token to token, so treat the curve as intuition, not a forecast.

Speculation helps most when the server is quiet. With a big batch, the arithmetic is no longer idle, and checking guesses that get rejected starts to cost real throughput. The EAGLE 3.1 team measured exactly this shape: per-user throughput rose 2.03× with one request in flight, 1.71× with four, and 1.66× with sixteen [12].

8. Quantization: fewer bytes per weight

If decode speed is roughly bandwidth ÷ bytes, then halving the bytes per weight roughly doubles it, and halves the memory too. Quantization stores weights as low-bit numbers plus a scale that maps them back:

The difficulty is outliers. One large weight stretches the scale, and every small weight near it lands on the same few levels, often zero. Try it, then fix it:

That’s why practical schemes give each small group of weights its own scale, as the second checkbox does. The best-known methods also decide which weights to protect. GPTQ rounds weights one at a time and adjusts the rest to cancel the error [18]. AWQ observes that protecting about 1% of salient weights, picked by looking at the activations rather than the weights, removes much of the error, and it protects them by scaling their channels instead of keeping them in higher precision [19]. Hardware now builds the group idea in: NVIDIA’s NVFP4 format stores 4-bit values with an 8-bit scale for every 16 of them, plus one scale per tensor. That’s 4.5 bits per weight. NVIDIA reports accuracy within about one point of fp8 on DeepSeek-R1 [20].

What you quantize decides what you gain:

• Weights speed up decode and let big models fit on small hardware, because decode is mostly reading weights.
• Weights and activations (for example fp8 throughout) also speed up prefill, on GPUs whose tensor cores do low-precision math natively: an H100’s fp8 peak is twice its bf16 peak [4].
• The KV cache (for example fp8) fits longer contexts or more users, and cuts the cache reads in long-context decode. Toggle it in the §6 figure. This is KV cache quantization, and it attacks the one operation in §3 that batching can’t help.

9. Pruning: fewer weights

Pruning removes weights instead of shrinking them. Removing them is the easy part; getting the hardware to actually skip them is harder.

• Unstructured. Zero the smallest weights wherever they are. Very large models tolerate this well: SparseGPT pruned OPT-175B and BLOOM-176B to 60% unstructured sparsity with a negligible rise in perplexity, removing over 100 billion weights [21]. But a GPU still multiplies the scattered zeros, so the speedup rarely shows up.
• 2:4 semi-structured. In every group of four weights, exactly two are zero. Sparse tensor cores, introduced with NVIDIA’s Ampere GPUs, exploit this pattern for twice the math throughput of dense units [22].
• Structured. Remove whole neurons, attention heads or layers. You get a smaller dense model that runs fast anywhere, but it costs the most accuracy, so it’s usually followed by distillation: the original model teaches the smaller one by having it imitate its outputs [24].

Here is the 2:4 pattern in practice: Red Hat pruned Llama 3.1 8B to 2:4 and kept 97–98% of its accuracy on two benchmark suites. Combined with 4-bit weights, it decoded 2.1–3.0× faster for a single user on A5000, A6000 and A100 GPUs [23]. Compare that with what the bytes alone predict:

The byte arithmetic says about 4.9× for 2:4 plus 4-bit. The measurement says 2–3×. The gap is everything that doesn’t shrink with the weights: activations, attention, the KV cache and kernel overheads. Keep that gap in mind whenever someone quotes a compression ratio as a speedup.

10. Beyond one GPU: splitting the model

Llama 3.1 70B in bf16 is 141 GB, and 405B is about 810 GB. Neither fits on one 80 GB GPU, so the model has to be split. There are four common ways, and real deployments combine them [32][34]:

• Tensor parallelism (TP) splits every layer’s weight matrices across GPUs, so each GPU holds a slice of every layer and they all work on the same token at once [33]. Each GPU reads only its share of the weights, so their bandwidth adds up and decode gets faster. The price is communication: the GPUs combine partial results inside every layer, so TP wants the fast links within one server (NVLink) [34].
• Pipeline parallelism (PP) gives each GPU or server a contiguous slice of layers. Only activations pass between stages, so it tolerates slower links, but one token still has to visit every stage in turn.
• Data parallelism (DP) runs independent copies of the model and spreads requests across them. It adds throughput, not room.
• Expert parallelism (EP) places different experts of a mixture-of-experts model on different GPUs.

vLLM’s own rule of thumb: use tensor parallelism inside a server, and add pipeline parallelism across servers when the model doesn’t fit in one [34]. Try it with the bigger models:

Notice how much the KV cache adds once you serve many long conversations, and how quantization (§8) can cut a whole server out of the plan.

11. Inference engines: putting it together

You rarely build any of this yourself. An inference engine bundles the batching, cache management, kernels and parallelism from this post behind an API. Three open-source engines cover most needs, and each is strongest at a different job:

My rule of thumb: start with vLLM for a GPU server, try SGLang when your requests reuse long shared prefixes, and use llama.cpp when the model has to run on hardware you already own. Then measure on your own workload, because the gaps between engines change with every release.

12. Where this simple model breaks

The figures use one idea, bytes versus FLOPS, with assumed efficiencies of 80% of bandwidth and 60% of peak FLOPS. That gets the shape of every curve right, but real systems differ in ways worth knowing:

• Overheads. Beyond kernel launches (§3), scheduling, sampling and detokenizing all take time the roofline ignores.
• Mixture-of-experts models. Each token uses only some experts. GLM-5.2 has 744B parameters but about 40B are active per token [17]. At batch 1 you read only the active experts, but a large batch touches many experts, so the bytes per step grow with the batch.
• Long contexts. With long conversations and big batches, reading the KV cache can rival reading the weights: for Llama 3.1 8B, sixteen 8K-token conversations hold more cache than the model has weights (§6). Then attention kernels, cache precision and context limits matter more than weight quantization.
• Many GPUs. Splitting a model across GPUs adds their bandwidth together, but the communication in §10 eats into it, more so across servers.
• Quality isn’t free. Aggressive quantization or pruning can cost accuracy, and benchmark averages can hide losses on your task, so measure. Speculative decoding is exact only if the acceptance rule is implemented exactly.
• Determinism isn’t free either. Batch-invariant kernels make answers reproducible, at a real cost in speed today [13].

My view: the simple model is still the right first tool. It tells you which wall you’re hitting before you spend a week tuning the wrong thing.

13. A checklist for a slow deployment

Start from the symptom, not the technique.

1. Measure first. Record TTFT and TPOT at the median and the 99th percentile, plus throughput and goodput under realistic load. llama bench and vllm bench both do this [1][2].
2. Slow first token? Look at prompt length. Turn on prefix caching if prompts share a beginning, and chunked prefill so long prompts don’t block short ones. At large scale, consider disaggregating prefill and decode.
3. Slow streaming with few users? You’re memory-bound. Quantize the weights, then try speculative decoding; n-gram drafting is nearly free when answers quote the prompt, as in code edits or RAG.
4. Low throughput when busy? Grow the batch. That usually means freeing memory for the KV cache: fp8 cache, a smaller maximum context, or quantized weights.
5. Out of memory, or requests queueing? Work out the KV budget from §6, then the GPU count from §10, before buying hardware.
6. Small model, small batch, and the GPU still looks idle? Suspect overhead (§3). Check that CUDA graphs or an equivalent are on.
7. Need reproducible outputs? Use batch-invariant kernels and budget for the slowdown.
8. After any compression, re-run your own evals. Benchmarks from a vendor’s blog are a starting point, not a guarantee for your task.

14. Where to read next

If you want to go deeper, here is the order I’d take, with the best resource I know for each step:

1. The model. Brendan Bycroft’s LLM visualization walks through a transformer in 3D [35]. Hugging Face’s KV Cache from scratch shows prefill, decode and the cache in code [14].
2. The hardware. Horace He’s Making Deep Learning Go Brrrr is the essay on compute, bandwidth and overhead [26]. Damek Davis’s Basic facts about GPUs adds arithmetic intensity, fusion and tiling [31].
3. The math of serving. All About Transformer Inference, from Google DeepMind’s How to Scale Your Model, works through the roofline argument of §3–4 in full [3].
4. The techniques. NVIDIA’s Mastering LLM Techniques: Inference Optimization is a compact survey of everything from batching to parallelism [32]. Then read the original papers: PagedAttention [6], FlashAttention [27] and speculative decoding [8].
5. A real engine. Aleksa Gordić’s Inside vLLM follows one request through a production system [2]. For a sharp lesson in what batching does under the hood, read Horace He’s Defeating Nondeterminism in LLM Inference [13].
6. At scale. The GLM-5.2 write-up shows disaggregation, 4-bit weights and speculative decoding working together on 24 GPUs [17].

For reference, keep Modular’s LLM Inference Handbook open [25].

Cite this post

@misc{moniruzzaman2026llminference,
  author       = {Monir Moniruzzaman},
  title        = {Interactive Guide to LLM Inference},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://monirzaman.github.io/interactive-guide-to-llm-inference/}}
}

References

[1] llama.app. Prefill vs. Decode. llama.cpp documentation, accessed October 2026.

[2] A. Gordić. Inside vLLM: Anatomy of a High-Throughput LLM Inference System. vLLM Blog, September 5, 2025.

[3] J. Austin, S. Douglas, R. Frostig, A. Levskaya, C. Chen, S. Vikram, F. Lebron, P. Choy, V. Ramasesh, A. Webson, R. Pope. All About Transformer Inference. In How to Scale Your Model, Google DeepMind, February 4, 2025.

[4] NVIDIA. H100 Tensor Core GPU specifications. H100 SXM: 80 GB, 3.35 TB/s, 1,979 BF16 TFLOPS with sparsity (989 dense). Accessed October 2026.

[5] Llama Team, AI @ Meta. The Llama 3 Herd of Models. arXiv 2407.21783, 2024.

[6] W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

[7] G.-I. Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.

[8] Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.

[9] C. Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv 2302.01318, 2023.

[10] Y. Li et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv 2401.15077, 2024.

[11] T. Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv 2401.10774, 2024.

[12] EAGLE Team, vLLM Team, TorchSpec Team. EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec. vLLM Blog, May 26, 2026.

[13] H. He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines blog, September 10, 2025.

[14] A. R. Gosthipaty et al. KV Cache from scratch in nanoVLM. Hugging Face blog, June 4, 2025.

[15] J. Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.

[16] Y. Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.

[17] DaoCloud Team. From Day 0 to Production SLAs: Serving GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM. vLLM Blog, July 23, 2026.

[18] E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.

[19] J. Lin et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024.

[20] E. Alvarez, O. Almog, E. Chung, S. Layton, D. Stosic, R. Krashinsky, K. Aubrey. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference. NVIDIA Technical Blog, June 24, 2025.

[21] E. Frantar, D. Alistarh. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ICML 2023.

[22] A. Mishra et al. Accelerating Sparse Deep Neural Networks. arXiv 2104.08378, 2021.

[23] E. Kurtić, A. Marques, M. Kurtz, D. Alistarh, S. Pandit. 2:4 Sparse Llama: Smaller models for efficient GPU inference. Red Hat Developer, February 28, 2025.

[24] S. Muralidharan et al. Compact Language Models via Pruning and Knowledge Distillation. arXiv 2407.14679, 2024. See also NVIDIA’s walkthrough, How to Prune and Distill Llama-3.1 8B to an NVIDIA Llama-3.1-Minitron 4B Model.

[25] Modular. LLM Inference Handbook. Continuously updated, accessed October 2026.

[26] H. He. Making Deep Learning Go Brrrr From First Principles. 2022.

[27] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.

[28] L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. arXiv 2312.07104, 2023 (v2, 2024).

[29] ggml-org. llama.cpp. GitHub repository, accessed October 2026.

[30] NVIDIA. NVIDIA Hopper Architecture In-Depth. NVIDIA Technical Blog, March 22, 2022.

[31] D. Davis. Basic facts about GPUs. June 18, 2025.

[32] S. Verma, N. Vaidya. Mastering LLM Techniques: Inference Optimization. NVIDIA Technical Blog, November 17, 2023.

[33] M. Shoeybi et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv 1909.08053, 2019.

[34] vLLM. Parallelism and Scaling. vLLM documentation, accessed October 2026.

[35] B. Bycroft. LLM Visualization. Interactive 3D walkthrough, accessed October 2026.

[36] A. Vaswani et al. Attention Is All You Need. NeurIPS 2017.

[37] Anthropic. Prompt caching. Claude API documentation, accessed October 2026.