Quantization is now table stakes. Any team running a large language model in production in 2026 that hasn’t applied INT8 weight quantization is leaving throughput and memory headroom on the floor. The problem isn’t whether to quantize — it’s that teams often reach for quantization as a first and only move, then wonder why inference costs are still unsatisfying. The real discipline is sequencing: applying the right optimization at the right point in the stack, against the workload profile that actually governs your deployment costs.

Naive quantization no longer differentiates engineering teams. The bottleneck has shifted upstream — to batching strategy, KV cache management, and, for latency-constrained deployments, speculative decoding. These techniques interact, and applying them in the wrong order can neutralize gains or introduce new bottlenecks. Earlier coverage on this site surveyed the individual techniques in the inference optimization toolkit — quantization, batching, KV cache, distillation, speculative decoding; this piece works through the sequencing question those techniques raise once a team has all of them available: which order produces the best results, and what workload dependencies determine which lever matters most.

Establish the Constraint Before Reaching for a Technique

Every optimization decision in inference engineering resolves to three competing constraints: latency, throughput, and cost. Improving one usually worsens at least one other. Before touching the stack, teams need a stable characterization of which constraint is binding.

Throughput-constrained workloads — batch jobs, asynchronous pipelines, embedding generation, high-volume classification — can tolerate latency in exchange for higher requests-per-second on fixed hardware. The optimization priority is utilization: keeping accelerators busy.

Latency-constrained workloads — real-time chat, interactive code completion, low-percentile-latency SLA commitments — cannot accept throughput gains that come at the cost of tail latency. The optimization priority is per-request speed, which points toward different techniques.

Memory-constrained deployments — fitting a given model class on available hardware — have a different primary lever: reducing the memory footprint of weights and the KV cache enough to enable serving at all, then layering other optimizations on top.

Getting this classification wrong leads to applying techniques that address the wrong bottleneck. A team that applies aggressive quantization to a latency-constrained deployment may improve throughput on paper while worsening p99 latency in production. The cost–latency–quality triangle is the central trade-off framework: pick two to optimize.

Layer 1: Batching Strategy Is the First Lever

Before quantizing weights or tuning caches, fix the batching strategy. The reason is straightforward: a GPU sitting at 30% utilization while waiting for requests is unoptimizable at any other layer — you’re paying for compute you’re not using.

Static batching holds all requests in a batch until the longest sequence finishes generating. This is implementation-simple but produces significant idle time: a request that finishes generating after 50 tokens holds up all other batch members until the 500-token request completes. For generative models, static batching is almost always the wrong default.

Continuous batching — also called iteration-level scheduling or in-flight batching — schedules at each decode step. Requests that finish early release their slots immediately; waiting requests fill them on the next iteration. In typical production workloads, continuous batching improves GPU utilization by 2–5× over static batching on generative workloads. It is supported natively by vLLM, SGLang, and TensorRT-LLM, the serving engines that high-performance teams have converged on.

Apply continuous batching first. It changes the utilization baseline against which all subsequent optimizations are measured. Quantization gains look very different on a GPU at 75% utilization than on one at 30%.

Layer 2: Quantization — Match Precision to Hardware and Task

Once batching is optimal, quantization is the primary lever for reducing the memory bandwidth costs that dominate large model inference. The choice of precision target is not arbitrary.

INT8 weight quantization is the safe default. It reduces model weight memory by roughly 2× compared to FP16, improves memory bandwidth throughput accordingly, and typically produces negligible quality degradation on most benchmark categories. LLM.int8() (arXiv:2208.07339) demonstrated that 8-bit weight quantization can be applied without significant degradation by handling outlier features in a separate FP16 path.

INT4 weight quantization via GPTQ or AWQ compounds the memory reduction to roughly 4× but introduces measurable benchmark degradation — typically 1–3 points on standard evals. AWQ (Activation-aware Weight Quantization) has become the standard choice for production INT4 deployment because it identifies and protects “salient” weights that disproportionately affect output quality. For memory-constrained deployments, INT4 AWQ + KV cache quantization + PagedAttention can fit a 70B-parameter model on hardware that previously required four GPUs for FP16 serving.

FP8 quantization is optimal for Hopper and Blackwell GPU architectures, offering better quality-throughput trade-offs than INT8 on hardware that natively supports FP8 arithmetic. For older GPU generations, INT8 with SmoothQuant remains the recommended path.

The ordering matters: quantize weights before addressing KV cache, because weight quantization determines the baseline memory footprint that governs how much KV cache you can sustain at a given concurrency level.

Validate quantization on your own task-specific test set before deploying. A model that loses 1.5 points on MMLU may lose 8 points on the specific code generation or structured output task you’re serving — the aggregate benchmark won’t catch that.

Layer 3: KV Cache Management

The key-value cache is the mechanism that enables autoregressive generation without recomputing attention over all prior tokens on every step. It is also the primary memory pressure point for long-context workloads and high-concurrency deployments.

Paged attention, popularized by vLLM, applies virtual memory concepts to KV cache allocation. Rather than pre-allocating a contiguous maximum-length block per request, it allocates fixed-size pages on demand. This eliminates internal fragmentation and allows substantially more concurrent requests to share GPU memory. For deployments with variable-length requests — which is most deployments — paged attention typically reduces KV cache memory waste by 50–80%.

Prefix caching is the highest-leverage KV cache optimization for applications with shared system prompts. If thousands of requests per hour share a 2,000-token system prompt, caching the KV for that prefix means the attention computation for those tokens runs once rather than once per request. The savings scale linearly with shared prefix length and request volume.

KV cache quantization adds another compression layer. Quantizing KV entries to FP8 or INT8 reduces cache memory by 2× with modest quality impact; INT4 KV quantization achieves 4× reduction. Apple’s QuantSpec paper demonstrates a self-speculative decoding framework using hierarchical 4-bit quantized KV cache, achieving acceptance rates above 90% and consistent end-to-end speedups around 2.5×. KV cache quantization is orthogonal to most other cache management techniques and composes cleanly with paged attention.

For long-context workloads (document processing, multi-turn dialogue with extended histories, retrieval-augmented pipelines), KV cache memory pressure is often the limiting factor before compute becomes the constraint. Cache eviction policies — which requests lose their KV entries when memory is exhausted — need to be tuned against your workload’s access patterns, not left at framework defaults.

Layer 4: Speculative Decoding for Latency-Constrained Deployments

Speculative decoding exploits an asymmetry in the computational cost of autoregressive generation: verifying a token is faster than generating one. A small, fast draft model proposes several tokens ahead; the large target model verifies the entire draft in a single forward pass. Accepted tokens are kept; rejected tokens trigger standard decoding. Quality is unchanged — rejected tokens are never used.

The speedup depends on the acceptance rate, which depends on how well the draft model’s distribution matches the target. On structured outputs and code generation — where the target model’s next-token distribution is relatively concentrated — acceptance rates are high and speculative decoding consistently delivers 1.5–2.5× throughput improvement. On open-ended generation with high entropy outputs, acceptance rates fall and gains approach zero.

Speculative decoding addresses a different bottleneck than quantization. Quantization reduces memory bandwidth costs; speculative decoding reduces the effective number of serial generation steps per request. The two are orthogonal and compose — QuantSpec applies both simultaneously, quantizing both the draft model’s KV cache and weights while using it for speculation.

The operational constraint is that running both a draft model and a target model simultaneously requires additional memory headroom. For deployments already near memory limits after quantization, speculative decoding may require a hardware upgrade. Apply it after the memory footprint from quantization and KV cache management is stable.

Combining the Stack: Sequencing in Practice

The ordering that tends to maximize returns:

  1. Fix batching strategy (continuous batching as default for generative workloads).
  2. Apply weight quantization appropriate to the hardware (FP8 on Hopper/Blackwell, INT8 or INT4 AWQ elsewhere), validated against your task distribution.
  3. Add paged attention for KV cache allocation; enable prefix caching if shared system prompts are present.
  4. Apply KV cache quantization if memory pressure remains after weight quantization.
  5. Add speculative decoding if latency is the primary remaining constraint and memory headroom allows.

The compounding math is real: a Llama-3-70B model that requires four A100 80GB GPUs in FP16 with static batching can be served on a single GPU with INT4 quantization, paged attention, and KV cache quantization, with continuous batching enabling the concurrency needed to justify the hardware. That is not a theoretical result — it reflects what production teams have achieved with current frameworks.

What the Stack Doesn’t Fix

Optimization techniques improve efficiency at a given model size and architecture. They don’t substitute for selecting the right model for the task. A 70B model serving a task that a well-distilled 7B model handles equally well is not a quantization problem — it’s a model selection problem. Tiered routing, where simple requests go to cheaper small models and harder requests escalate to larger ones, often produces larger cost reductions than any single optimization technique applied to a uniformly large model.

Similarly, autoscaling inference endpoints to match traffic demand addresses fleet-level utilization that per-request optimization cannot reach. A deployment optimized at the per-request level but running at 40% average fleet utilization due to bursty traffic is still over-provisioned. Right-sizing the fleet against traffic patterns is the operational layer that sits above the inference optimization stack.

Multi-agent deployments add a layer above that again: a well-optimized inference stack under a single model call does nothing to prevent orchestration-level cost blowouts — unbounded retry loops, cascading timeouts, retries that run until a circuit breaker trips — that occur one level up in the system, in the control flow between agent calls rather than inside any individual inference request.

Frequently Asked Questions

What order should inference optimizations be applied?

Start with batching strategy — continuous batching for generative workloads. Then apply weight quantization (INT8 safe default; INT4 AWQ for memory-constrained deployments). Add paged attention and prefix caching for KV cache management. Apply KV cache quantization if memory pressure remains. Add speculative decoding last if latency is the binding constraint and memory allows. Applying in this order ensures each layer is measured against a stable baseline.

Does INT4 quantization meaningfully degrade model quality?

On standard aggregate benchmarks, INT4 typically shows 1–3 point degradation. Whether that matters depends on the task. Structured output, code generation, and domain-specific tasks can show larger degradation than general-knowledge tasks. Always validate on a task-specific held-out test set before deploying quantized models to production; aggregate benchmark scores are not a reliable proxy for task-specific performance.

When does speculative decoding help and when does it not?

Speculative decoding helps most when the target model’s output distribution is predictable — code generation, structured JSON output, task-specific pipelines with constrained outputs. It helps less for open-ended generation with high entropy. The critical variable is the acceptance rate: above roughly 70%, speedups are meaningful; below 40%, the overhead from draft model inference and rejected tokens erodes gains.

Can quantization and speculative decoding be combined?

Yes, and they compose cleanly. QuantSpec demonstrates this explicitly: a self-speculative decoding framework using quantized KV cache and quantized draft model weights achieves acceptance rates above 90% and consistent 2.5× end-to-end speedup. The practical requirement is sufficient memory headroom to run both models simultaneously, which quantization itself helps achieve.

What is prefix caching and when is it valuable?

Prefix caching stores the key-value representations of a shared prompt prefix and reuses them across requests. It is valuable whenever many requests share a long common prefix — typically a system prompt, few-shot examples, or a document preamble. The savings scale with prefix length and request volume; for applications with a 2,000-token system prompt serving thousands of requests per hour, prefix caching can reduce KV computation by the corresponding proportion for every request that hits the cache.