The Structure of LLM Inference Revolutions

TL;DR: LLM inference is undergoing a paradigm shift from batch parallelism to sequence parallelism.

In The Structure of Scientific Revolutions, Thomas Kuhn argued that science does not advance via a mere accumulation of discoveries.¹ Researchers work within a shared set of assumptions about what the important problems are and how to solve them: a paradigm. I believe we are in a paradigm shift in LLM inference. We are moving from batch parallel decoding to sequence parallel decoding.

Kuhn’s account of scientific progress can be organized into five phases.²

1. Pre-paradigm. Competing approaches have yet to settle into a shared framework.

2. Normal science. Researchers solve problems within the prevailing paradigm.

3. Anomalies. Results put the paradigm’s assumptions under pressure.

4. Paradigm shift. Researchers reexamine those assumptions and develop a new framework.

5. Post-revolution. The new framework becomes the basis for normal science.

Pre-paradigm

LLM serving inherited infrastructure built for general-purpose deep learning. Production systems such as TensorFlow Serving³ and NVIDIA Triton Inference Server⁴ handled model deployment, request batching, and efficient execution across machine-learning frameworks. Batching requests in this time was already common practice. However, the advent of autoregressive LLM inference posed new challenges. One request requires repeated model executions, retains a growing state (the KV cache), and finishes after a variable number of tokens.

It became clear that autoregressive decoding could be memory-bandwidth-bound. Each step reads model weights and the KV cache to generate only one new token per request. Pope and colleagues⁵ explain how this bottleneck changes depending on the workload. At small batches and short contexts, loading weights dominates memory traffic. Batching spreads the weight-loading cost across more requests, but each request still has its own KV cache to read. As batches and contexts grow, those cache reads can become the dominant memory cost.

Normal science

The response to the weight-loading bottleneck was batch parallelism. In this blog, I will examine the following paradigmatic assumption: batch parallelism is the best way to maximize throughput in an LLM serving system. Under Kuhn’s framework, what then proceeds in this paradigm is normal science. Several natural questions start to emerge: How do we keep the batch full? How do we fit more requests in memory? How do we keep other work from delaying decoding?

Batching amortizes weight reads across tokens from independent requests. A batch of B sequences allows a dense linear layer to perform roughly B times as much computation per weight load, turning matrix–vector computation into denser matrix–matrix computation. This raises arithmetic intensity and accelerator utilization. In the context of LLM serving, the systems problem is about maintaining a sufficiently large active batch despite having variable request lengths and memory occupied the KV cache of each request.

Orca⁶ introduced iteration-level scheduling so completed requests could leave and new requests could join between decoding steps. vLLM’s PagedAttention⁷ reduced KV-cache fragmentation, allowing more requests to fit in memory. SGLang⁸ developed KV-cache reuse across generation calls, avoiding redundant computation and storage for shared prefixes.

These advances made it easier to sustain large batches and use their memory efficiently. Kwon and colleagues state the premise in the opening of the vLLM paper⁷.

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size.

In LLM inference (and basically all systems), the goal is to manage two key metrics: (1) System throughput, which measures how many tokens the whole server produces per second (2) Interactivity, which measures how quickly each user receives tokens. A larger batch often improves (1) at the cost of (2).

Once a system Pareto dominates another system on the throughput-interactivity tradeoff, it becomes very hard to ignore. In fact, if you hold economic cost and token quality constant, it is completely irrational not to switch to the strictly more efficient system. Note that Pareto dominate means always better in both metrics, so if a system can hold onto even a single non-dominated point, there can still be doubt about switching.

Anomalies

Under a scientific paradigm, a series of anomalies can accumulate. For example, Newtonian predictions for Mercury’s orbital precession fell 43 arcseconds per century short of observations. It was hypothesized at the time that a hidden planet named Vulcan existed closer to the Sun and was pulling on Mercury, but decades of searching turned up nothing.¹⁰ In an analogous manner, several anomalies call the batch parallel paradigm into question, but in each case it is always possible to explain it away.

Sequence parallel decoding was first introduced by speculative decoding¹¹. A small draft model proposes a continuation and a target model verifies the proposed tokens in parallel. This did not disrupt the batching paradigm, because it was generally believed to only be helpful in high-interactivity, low batch size setting. For example, SmartSpec¹² found that speculation could make interacticity worse under higher request rates or low acceptance. Su and collaborators¹³ found that longer speculation block size could slow decoding at larger batches. SPEED-Bench¹⁴ explained that as batching moves execution toward a compute-bound regime, the spare capacity that speculative decoding can exploits diminishes. As summarized by the MagicDec abstract¹⁵ with regard to speculative decoding, “the conventional wisdom suggests that its efficacy is limited to small batch sizes.”

However, MagicDec was an early counterexample to the batch parallel paradigm. Its experiments showed that speculation could become more effective as the batch grew. Figure 3 holds the model, hardware, and drafting settings fixed and changes the context length. At 4k token context, larger batches reduce the speculative decoding speedup, but at 32k, the larger batches increase speedup. However, one can be dismissive because it only applies to long-context scenarios.

These results are not immediately intuitive. It should be easier to parallelize over independent users compared to highly interdependent tokens within a sequence. The reason these results are possible is because batch parallel decoding grows the KV cache in the number of users, and the KV cache can eventually become the bottleneck on throughput. Sequence parallel decoding makes it possible to increase throughput without adding more KV pressure. Put another way, speculative decoding can increase high-concurrency throughput by amortizing the KV cache footprint across multiple accepted tokens per step.

Figure 4 compares the number of tokens MagicDec generates per verification with the number needed to match autoregressive decoding. As context grows, the tokens per verification cycle stays roughly constant at 2.5, while the acceptance rate needed to beat pure autoregression falls. For this particular model, the interpolated crossover point where sequence parallelism achieves better throughput is just over 16K context length.

MoESD¹⁶ found another example of sequence parallelism succeeding in sparse mixture-of-experts models. Each token uses only a few experts, so a batch can have tokens that activate many experts. At a moderate batch size, a decoding step may already require reading nearly all the expert weights, even though each expert processes only a few tokens. Verifying several proposed tokens per request increases arithmetic intensity with little additional weight-reading cost. At sufficiently high acceptance rate, this increases throughput. At larger batches, computation eventually becomes the bottleneck and speculative decoding achieves diminishing returns.

The EAGLE family improved speculation by giving the drafter access to the target model’s internal features. The original EAGLE¹⁷ predicts the target’s next hidden feature vector from its final-layer features and the sampled token, then uses the target’s output layer to turn that prediction into token probabilities. EAGLE-3¹⁸ combines features from early, middle, and late layers, and trains on its own previous draft outputs so it can continue drafting accurately after the first step. In the experiment below, EAGLE-3 obtains a strong advantage over autoregression in even high concurrency settings.

While speculative decoding was producing these anomolies, an independent line of research was developing another way to parallelize over the sequence. A key advance was Austin and colleagues’ work on discrete diffusion in 2021¹⁹, which extended diffusion to discrete data. A model learns to reverse a noise corruption process, allowing it to generate text by updating multiple positions in parallel over successive denoising steps. MDLM in 2024²⁰ further narrowed the gap with autoregression, and LLaDA in 2025²¹ scaled the approach to an 8B model with instruction-following capabilities. Thus, sequence parallel inference was of growing interest to the research community.

While diffusion models presented opportunities for high interactivity, early open models often struggled to turn parallel generation into higher throughput.²⁴ Furthermore, they are trained completely differently from autoregressive models, and it still is not clear whether diffusion models can usurp autoregressive models in terms of output quality, so from the standpoint of LLM inference paradigms, they could still be dismissed.

Paradigm shift

This is the phase I believe we are currently in. In my opinion, speculative diffusion (using a diffusion drafter in speculative decoding) produces results that can no longer be dismissed.

I believe diffusion drafters are fundamentally better than autoregressive drafters because they can achieve higher acceptance rates in a fixed amount of drafting time. Consider a larger or deeper draft model that can make better proposals. An autoregressive drafter uses the full depth of the model for every token it proposes. This creates pressure to keep the drafter shallow even when more capacity could improve acceptance rate. A diffusion drafter can instead do parallel computation on many proposed positions simultaneously, allowing a larger drafter to fit within the same latency budget. Parallel drafting makes it possible to scale the drafter, which can increase the number of accepted tokens per verification.

With this in mind, DFlash²⁵ proposes a block of tokens using diffusion and gives every draft layer access to the target model’s internal features, similar to EAGLE. Unlike EAGLE, they show that diffusion drafting latency can remain constant in the number of tokens drafted.

Uno²⁷ builds upon their results, using the base model’s weights with a diffusion adaptation for drafting and sharing the KV cache with verification. We can examine the corresponding throughput–interactivity frontier.

As I stated previously, if one method dominates another on throughput-interactivity, it will be impossible to argue not to change to that method. A fair critique is that the above results are only on an 8B model. Scaling the model parameters and using state of the art architectures perhaps increases compute intensity faster than it increases KV cache pressure. If so, batch parallelism would be favored in those regimes. Fortunately, SemiAnalysis’s InferenceX provides comparisons of large open source models with speculative decoding turned on and off on various hardware configurations. Note that these include several speculative methods, not just diffusion drafting.

Thus, even for large models there is substantial evidence that sequence parallelism can improve throughput over batch parallelism alone. As diffusion drafters gain popularity and improve over time, I anticipate eventually that the highest throughput configuration at any level of interactivity will include some form sequence parallelism through speculative decoding.

Another work worth mentioning is DSpark²⁸. DSpark obtains even more improvement by using a special decoding head to improve causal consistency of diffusion drafts. Slight tangent, but I must shout out my labmates for exploring this idea before DSpark in Breaking the Factorization Barrier in Diffusion Language Models.²⁹ Anyway, DeepSeek deploys speculative diffusion in its V4 serving system²⁸, replacing its previous single-token MTP setup. Under live user traffic, it improved the measured throughput–interactivity frontier.

Post-revolution

I would consider the revolution complete when all major inference providers routinely use sequence-level parallelism to achieve both higher throughput and interactivity. At that point, it becomes ingrained as an assumption of LLM serving design, and the field returns to normal science within the new paradigm.

There will still be plenty of scientific questions in the new paradigm. For example, I personally do not think speculative decoding is very clean. There is a drafter that spends some amount of compute and a verifier that spends another amount. Why should there be exactly two levels? Why not a hierarchy of adaptive computation. There ought to be a different amount of computation allocated according to the difficulty of the prediction. The right abstraction might allow infinitely many possible compute budgets rather than a fixed small model paired with a fixed large one.

Another possibility is that there should simply be one large model. Why not pure diffusion? Diffusion is computationally similar to speculative decoding in the sense that a forward pass proposes tokens and a subsequent forward pass accepts or rejects and refines the proposals. Google’s DiffusionGemma³⁰ achieves high throughput and high interactivity.

However, critiques will argue that diffusion has yet to match autoregression in quality. In addition to improving output quality, I would also like to see a diffusion model designed that outright dominates autoregressive models and speculative decoding at even the highest concurrency settings. This has not been achieved yet, and there is still a lot of room for exploration.

What will emerge in a sequence-parallel decoding paradigm?

References