Running a model in production is a different discipline from training one. Training is a one-time capital expenditure — inference is the recurring operating cost that compounds every time a user submits a request. A model serving a million queries a day at $0.002 per request burns $730 a year; scale to ten million queries and that number becomes material. The techniques that reduce inference cost without degrading the user experience are not exotic research ideas — most are production-ready and widely deployed.

The problem is that the landscape of optimization choices is genuinely complex. Decisions interact: aggressive quantization saves memory bandwidth but may require retuning; continuous batching improves throughput but increases tail latency; distillation cuts compute but requires a training run. This article is an attempt to give ML systems engineers a realistic map of the trade-off space, with enough technical specificity to make real engineering decisions.

We’ll work through the major levers in roughly the order you would reach for them in a real deployment.

What Does “Inference Cost” Actually Mean?

Before optimizing, you need a stable definition of cost. The two most useful metrics are cost per token (for generative models) and cost per request (for classification or embedding workloads). Both are derived from the same inputs: compute time, memory usage, and the amortized cost of the hardware.

On cloud infrastructure, inference cost maps directly to GPU-hours or accelerator-hours. On owned hardware, it maps to depreciation plus power plus opportunity cost. In either case, the levers are the same: reduce compute per token, increase utilization of the hardware you have, or move to cheaper hardware without unacceptable quality loss.

Latency is a different axis entirely. A batch job can tolerate high latency in exchange for high throughput; a real-time chat application cannot. The cost–latency–quality triangle is the central constraint of inference engineering: you can optimize for any two of the three, but all three simultaneously is rarely achievable. Being explicit about which vertex matters most for a given workload is prerequisite to choosing the right technique.

Quantization: Trading Bits for Throughput

Quantization reduces the numerical precision of model weights and, optionally, activations. The most common targets are INT8 (8-bit integers) and INT4 (4-bit integers), down from the FP16 or BF16 used during training.

Weight-only quantization compresses stored weights to INT4 or INT8 while keeping activations in FP16 during computation. This reduces memory bandwidth — the dominant bottleneck for large models on GPU — and often improves throughput with minimal quality loss. The LLM.int8() paper by Dettmers et al. (arXiv:2208.07339) demonstrated that 8-bit weight quantization of large language models could be done without significant degradation by decomposing outlier features into a separate FP16 path.

Weight-and-activation quantization (sometimes called W4A8 or W8A8) is more aggressive. It quantizes activations at runtime as well, enabling integer arithmetic on hardware that supports it. The gains are larger, but so is the risk: activation distributions vary by input, and outliers can cause significant error if not handled carefully. Post-training quantization (PTQ) works without retraining but has limits; quantization-aware training (QAT) recovers much of the lost quality but requires a training run.

A practical rule of thumb: INT8 weight quantization is usually safe with no additional work. INT4 weight quantization (e.g., GPTQ, AWQ) requires calibration data and produces measurable quality degradation on benchmarks — whether that degradation matters for your application is an empirical question, not a theoretical one. For tasks with tight quality requirements (code generation, structured output, medical text), validate on your own test set before deploying.

Batching and Continuous Batching: Utilization Is Everything

A GPU sitting at 30% utilization while waiting for the next request is wasted capital. Batching is the mechanism that keeps accelerators busy.

Static batching groups multiple requests into a single forward pass. It is straightforward to implement and works well for fixed-length inputs (embeddings, classification). The problem with generative models is that requests in the same batch finish at different times. With static batching, the batch cannot release until the longest sequence completes, leaving accelerator capacity idle.

Continuous batching (also called iteration-level scheduling or in-flight batching) solves this. Rather than holding a batch together for the full generation, the scheduler operates at each decode step and can insert new requests as slots free up. A request that finishes early releases its slot immediately; a new request fills it on the next iteration. The result is substantially higher throughput on the same hardware — in typical production workloads, continuous batching can improve GPU utilization by 2–5× compared to static batching.

The trade-off is implementation complexity and slightly increased tail latency for long-running requests, since new requests competing for slots can cause brief scheduling delays. For most applications, continuous batching is the correct default and should be adopted before reaching for more exotic techniques.

KV-Cache Management: The Memory Bottleneck in Autoregressive Generation

Autoregressive generation is compute-inefficient by nature: each new token requires attending over all previous tokens. The standard mitigation is the KV-cache — storing the key-value projections from previous tokens so they don’t need to be recomputed. The KV-cache is effective but expensive: for a large model serving long contexts, the cache can consume gigabytes of GPU memory per request.

Paged attention (the approach popularized by vLLM) applies virtual memory concepts to KV-cache management. Rather than pre-allocating a contiguous block for each request’s maximum possible length, it allocates fixed-size pages on demand. This eliminates internal fragmentation and allows many more concurrent requests to share GPU memory. The practical effect is higher throughput at the same memory budget.

For deployments serving many short requests, KV-cache pressure is usually manageable. For long-context workloads (document summarization, multi-turn dialogue with long histories, retrieval-augmented pipelines like those described in our article on retrieval-augmented generation), KV-cache management becomes critical. Strategies include evicting cache entries for stale requests, using prefix caching for shared prompt prefixes (effective when a system prompt is reused across many requests), and offloading cache to CPU memory with async prefetch.

Prefix caching deserves special mention: if many of your requests share a long system prompt, caching the KV for that prefix and reusing it across requests can dramatically reduce both compute and memory pressure. The savings scale with prompt length and request volume.

Model Distillation: A Smaller Model That Punches Above Its Weight

Distillation is a training-time technique that produces a smaller “student” model that mimics the behavior of a larger “teacher.” The student is trained not just on ground-truth labels but on the soft probability distributions output by the teacher — these distributions carry more information than hard labels and help the student learn the teacher’s generalizations.

For inference cost, the benefit is straightforward: a model with half the parameters runs faster and uses less memory. The question is how much quality you give up.

The honest answer is: it depends entirely on the task. For narrow, well-defined tasks (sentiment classification, intent detection, short-form summarization on a specific domain), distillation can produce a model that matches or nearly matches a much larger teacher on the target distribution while running at a fraction of the cost. For broad, open-ended tasks, the quality gap is more significant.

The operational cost of distillation is a real training run, which requires compute, a curated dataset, and careful evaluation. This makes it a better fit for stable, high-volume workloads than for rapidly iterating products. If you are serving a specific function at scale — a code completion model for a particular language, an intent classifier for a support system — distillation is often the highest-leverage optimization available.

Distillation is complementary to quantization: you can distill to a smaller model and then quantize it, compounding the efficiency gains.

Speculative Decoding: Buying Cheap Tokens

Speculative decoding exploits an asymmetry: verifying a token is faster than generating one. The technique uses a small, fast “draft” model to propose several tokens ahead, then verifies the entire draft in a single forward pass of the large model. If the draft tokens are accepted, you have generated multiple tokens for roughly the cost of one verification pass. If they are rejected, you fall back to standard decoding.

The speedup depends on the acceptance rate, which depends on how well the draft model approximates the large model on the target distribution. On code and structured outputs — where the large model’s distribution is relatively predictable — acceptance rates are high and speculative decoding can deliver meaningful throughput improvements (1.5–3× in well-matched settings). On open-ended generation with high entropy, acceptance rates drop and the gains shrink.

The technique has zero quality impact by construction: rejected draft tokens are discarded and the large model’s output is always used for the final sequence. This makes it attractive for latency-sensitive applications where you cannot afford quantization quality loss but need to serve more requests on the same hardware.

The practical constraint is that you need to maintain and serve both a draft model and the target model simultaneously. This adds operational complexity and memory overhead. For deployments already running near memory limits, speculative decoding may not be feasible without a hardware upgrade.

Hardware and Accelerator Selection: Matching the Workload to the Silicon

The transformer architecture — discussed in detail in our article on transformer architecture — is dominated by two types of operations: matrix multiplications (linear layers, attention) and memory bandwidth-bound operations (loading weights and KV-cache). The optimal hardware depends on which is the bottleneck.

For large models with high batch sizes, compute is the bottleneck — high-FLOP accelerators win. For large models with small batch sizes (latency-sensitive, low-concurrency deployments), memory bandwidth is the bottleneck — accelerators with high HBM bandwidth relative to compute are preferred. For smaller models, the ratio shifts again.

CPU inference is viable for small models (sub-1B parameters) and batch workloads that tolerate latency. For anything larger, dedicated accelerators are required. Across the GPU landscape, the relevant comparison points are memory capacity (which determines maximum model size and KV-cache budget), memory bandwidth (which determines throughput on bandwidth-bound workloads), compute throughput (FLOPS at FP16/BF16/INT8), and interconnect bandwidth (for multi-GPU inference).

Right-sizing is often more impactful than switching hardware families. Running a 7B-parameter model on a GPU designed for 70B models is wasteful; a smaller accelerator at higher utilization is frequently cheaper. The discipline of matching model size, batch size, and hardware is where significant cost is left on the table in practice.

Autoscaling and Right-Sizing: Operational Levers

Hardware efficiency matters at the unit level; fleet utilization matters at the system level. Inference workloads are typically bursty — traffic spikes during business hours, drops overnight. Paying for peak capacity 24 hours a day to handle two-hour peaks is expensive.

Autoscaling inference endpoints to match demand is the operational mechanism that addresses this. The key parameters are the metrics you scale on (requests per second, GPU utilization, queue depth), the scale-up and scale-down latency (GPU instances take time to start; scale-up must anticipate demand, not react to it), and the minimum fleet size (zero-scaling to save cost can introduce unacceptable cold-start latency for interactive applications).

For cost-sensitive deployments, a combination of a small always-on fleet plus burst capacity from spot or preemptible instances is a common pattern. The trade-off is reliability: spot instances can be preempted, requiring the serving layer to handle interruptions gracefully.

At the model level, “right-sizing” means selecting the smallest model that meets quality requirements for each request type. Routing easy requests to a cheaper small model and hard requests to a larger model — sometimes called speculative or tiered routing — can reduce average cost significantly without degrading user-facing quality, provided the routing logic is reliable.

Frequently Asked Questions

Does quantization hurt model quality?

It depends on the precision level and the task. INT8 weight quantization typically produces negligible quality degradation on most benchmarks. INT4 quantization shows measurable degradation — usually 1–3 points on standard evals — which may or may not matter for a specific application. Always validate on a task-specific test set before deploying quantized models to production.

What is continuous batching and why does it matter?

Continuous batching (also called in-flight batching) schedules inference at the per-token-generation step rather than per-request. When one request in a batch finishes generating, its slot is immediately filled by a waiting request. This eliminates idle GPU time from mismatched sequence lengths and typically improves throughput by 2–5× over static batching on generative workloads.

What is speculative decoding?

Speculative decoding uses a small, fast draft model to propose several tokens ahead, then verifies the draft in one forward pass of the large model. Accepted tokens are kept; rejected ones trigger standard decoding. Because verification is cheaper than generation, throughput improves when acceptance rates are high. Quality is unaffected — rejected tokens are never used in the final output.

How do I measure inference cost per token?

Track wall-clock time per request, tokens generated per request, and the amortized hardware cost per unit time (GPU-hour rate for cloud, or depreciation plus power for owned hardware). Cost per token = (hardware cost per second × seconds per request) / tokens per request. For batch workloads, measure at the batch level and divide by total tokens. Monitor separately for prompt tokens and completion tokens, as their computational profiles differ.

When does model distillation make sense over other techniques?

Distillation is highest-leverage for high-volume, stable, narrow tasks where a training run is feasible. If you are serving millions of requests per day on a well-defined task (intent classification, domain-specific summarization), a distilled model can match teacher quality on your distribution at a fraction of the inference cost. For broad, general-purpose tasks or rapidly evolving products, quantization or hardware right-sizing typically offers faster returns.