A trained model is not a deployed model. The gap between a checkpoint that scores well on a held-out set and a system serving millions of requests within a latency budget is filled almost entirely by serving infrastructure — the layer responsible for queuing, batching, scheduling, and scaling inference work across hardware that is expensive, memory-constrained, and idle more often than anyone would like.

This piece works through how model serving systems are actually built: how requests get batched without violating latency guarantees, how autoscaling handles bursty load, how multiple models share a fleet of GPUs, and what the major serving frameworks trade off against each other. None of this is exotic — it is closer to distributed systems and operations research than to ML research — but it is where most engineering effort in a production ML system actually goes.

Why Serving Is a Different Problem Than Training

Training and serving optimize for different things; conflating them is a common source of poor architectural decisions.

Training is throughput-oriented and batch-scheduled. A training job processes a fixed, known dataset, and the only deadline is when the job finishes — hours or days away. Batch sizes are chosen for hardware utilization, not individual latency, because no individual request is waiting on a training step.

Serving is latency-oriented and demand-driven. Requests arrive continuously, at unpredictable rates, from callers waiting on a response. A serving system must decide, in real time, how to batch requests for hardware efficiency without making any single request wait too long — and handle load that varies by an order of magnitude between a weekday afternoon and a weekend night, without overprovisioning (wasted GPU spend) or underprovisioning (missed latency targets).

This tension — batch for efficiency, respond individually for latency — is the central design problem in every serving architecture below.

Request Queuing and the Batching Problem

A single inference request rarely uses a GPU efficiently. Modern accelerators are throughput machines, built to process many parallel operations per cycle, and a batch size of one leaves most of that parallelism unused. The fix is batching — grouping multiple requests into a single forward pass — but naive batching runs into a scheduling problem specific to autoregressive generation.

Static batching, the simplest approach, waits until either a fixed number of requests has accumulated or a timeout elapses, then runs them together as one batch. This works reasonably well for models with fixed, short inference times (e.g., a classifier). It works poorly for large language models, because generation length varies enormously across requests: a batch of ten prompts might finish after twenty tokens for eight of them and continue for three hundred tokens on the other two, and static batching holds the entire batch, GPU included, until the slowest sequence finishes. Short requests get stuck behind long ones, and average latency degrades badly under mixed workloads.

Dynamic (continuous) batching solves this by decoupling the batch from a fixed group of requests. Instead of batching at the request level, the scheduler batches at the iteration level: at each decoding step, any finished sequence is evicted from the batch and its slot is immediately filled by a new request from the queue. The Orca paper (Yu et al., 2022) introduced this iteration-level scheduling for transformer inference, and it is now close to standard practice in LLM serving. The effect is that GPU occupancy stays high regardless of how variable generation lengths are, and short requests no longer wait behind long ones inside the same batch.

Continuous batching interacts directly with memory management, because each sequence in flight needs its own key-value (KV) cache — the accumulated attention state that grows with sequence length. Naively allocating a fixed, worst-case-sized memory block per sequence wastes GPU memory on padding for sequences that turn out to be short. The vLLM project’s PagedAttention (Kwon et al., 2023) addresses this directly, borrowing the paging concept from operating system virtual memory: KV cache is allocated in fixed-size non-contiguous blocks rather than one contiguous region per sequence, which reduces fragmentation and lets the scheduler admit far more concurrent sequences for a given amount of GPU memory. The combination of continuous batching and paged KV cache allocation is the single biggest throughput lever in current LLM serving stacks — gains of several times the throughput of naive request-level batching are commonly reported, though the exact multiple depends heavily on request length distribution and hardware.

Request queuing sits upstream of batching: admission control (shedding load once the queue would blow latency budgets), priority scheduling (latency-sensitive traffic preempting best-effort traffic), and backpressure signaling so upstream callers degrade gracefully rather than piling up retries.

Autoscaling Under Variable Load

Batching solves efficiency within a fixed pool of hardware. Autoscaling solves the separate problem of how large that pool should be at any given moment.

The central trade-off is cold-start latency versus idle cost. A GPU-backed inference server is expensive to leave running when idle, but also expensive to spin up from cold. Loading a multi-billion-parameter model’s weights onto GPU memory, initializing CUDA contexts, and warming up any JIT-compiled kernels can take anywhere from several seconds to over a minute depending on model size and storage bandwidth. A traffic spike that requires scaling from one replica to five is not instantaneous; the new replicas are not useful until they finish loading, and requests arriving in that window either queue, get rejected, or get served by an overloaded existing replica.

Several mitigations are standard practice:

Keeping a warm pool. Maintaining a small number of idle-but-loaded replicas absorbs the first burst of a scale-up event while additional cold replicas spin up behind them, at the cost of continuous GPU spend for the warm capacity.

Predictive scaling. Where load has a known cyclical pattern (daily traffic curves, batch job schedules), scaling ahead of the predicted spike avoids reactive scaling entirely. This only helps when load is actually predictable; it does nothing for genuinely bursty or novel traffic.

Scaling on leading indicators rather than lagging ones. Autoscaling on GPU utilization alone tends to react too late, because utilization only rises after requests have already started queuing. Scaling on queue depth or a moving average of request rate reacts earlier, at the cost of being noisier and requiring more careful threshold tuning.

Model caching and snapshotting. Keeping model weights in host memory or a fast local cache rather than re-fetching from remote storage on every cold start, or using memory-mapped weight loading, reduces the cold-start penalty directly — which reduces how much the other mitigations need to compensate.

GPU utilization itself deserves a caveat: it is necessary but insufficient. A GPU can report high utilization while still being memory-bound rather than compute-bound — busy on memory transfers rather than useful floating-point work. Teams that autoscale purely on utilization percentages, without also tracking throughput and latency, often discover they’ve been scaling in response to memory-bandwidth saturation rather than genuine compute demand.

Multi-Model Serving and Adapter Swapping

Most production inference fleets serve more than one model, and a dedicated GPU pool per model is rarely economical — many models see low or spiky individual traffic that doesn’t justify dedicated capacity, even though aggregate traffic across all of them is substantial.

Model multiplexing addresses this by co-locating multiple models on shared hardware, with a serving layer routing each request to whichever replica currently holds the requested model in GPU memory, loading it on demand if none does. This works well when the model set is large relative to available GPU memory and per-model traffic is intermittent, but it introduces the same cold-start problem at the level of individual models — a request for a rarely used model may wait for its weights to load.

LoRA adapter swapping is a more specific and increasingly common pattern for serving many fine-tuned variants of a single base model. Low-Rank Adaptation (Hu et al., 2021) fine-tunes a small number of additional parameters — low-rank matrices injected into specific layers — while keeping the base model’s weights frozen. Because the adapter is small, often under 1% of the base model’s parameter count, a serving system can keep one copy of the base model resident on a GPU and swap in different adapters per request with comparatively little overhead. Dozens or hundreds of task-specific fine-tunes can then share a single base model deployment rather than requiring one full duplicate per variant. The practical limits are how many distinct adapters can be batched together in one forward pass, and the latency cost of swapping in an adapter that isn’t already resident.

The Serving Frameworks Landscape

Open-source and vendor frameworks have converged on overlapping feature sets, though they emerged from different starting points and retain different strengths.

General-purpose inference servers, such as NVIDIA Triton Inference Server, are framework-agnostic by design — they serve models trained in PyTorch, TensorFlow, ONNX, and other formats behind a common API, with configurable batching, multi-model support, and hardware backend abstraction. This suits heterogeneous model portfolios, at some cost in optimization depth versus a framework purpose-built for one model type.

LLM-specialized serving engines, of which vLLM is the most widely cited example, are built specifically around the memory and batching characteristics of autoregressive transformer generation — continuous batching and paged KV cache management, described above, are native to this category rather than bolted on. The trade-off is narrower applicability: these engines are optimized for one model family and inference pattern, and generally aren’t the right tool for a vision model or a classical ML model.

Framework-native serving tools, such as TorchServe, prioritize tight integration with a specific training framework’s model format and ecosystem, reducing export and deployment friction for teams already standardized on that framework, at the cost of being less optimized for cross-framework fleets.

Kubernetes-native model serving orchestration layers, such as KServe, sit at a different layer of the stack — they don’t replace an inference engine so much as wrap one, providing standardized deployment manifests, autoscaling integration (including scale-to-zero), canary rollout support, and a consistent inference API across whatever underlying runtime is configured per model.

No single category is strictly superior; the right choice depends on model diversity, existing infrastructure, and traffic patterns. Teams frequently run more than one simultaneously — an orchestration layer managing several inference engines behind it is a common pattern, not an edge case.

Observability for Serving Systems

A serving system that cannot answer “how is it performing right now” is not production-ready, regardless of how it performs under test load. Serving-specific observability requires metrics that generic application monitoring rarely captures.

Latency percentiles, not averages. Mean latency hides the tail, and the tail is what determines whether users experience the system as reliable. p50, p95, and p99 latency should be tracked separately, and for LLM serving specifically, time-to-first-token and per-token latency should be tracked as distinct metrics from total request latency — a request can have excellent time-to-first-token and still feel slow if per-token generation is sluggish, or vice versa.

Throughput, measured in the right unit. Requests per second is reasonable for classification or embedding workloads with roughly fixed per-request cost. For generative workloads, tokens per second is more meaningful, because request cost varies enormously with output length — a request-per-second number can look identical while serving very different amounts of actual generated content.

Queue depth and time-in-queue. These are leading indicators of an overloaded system, visible before latency targets are actually breached. A rising queue depth with stable per-request processing time indicates the system needs more capacity; a rising per-request processing time with stable queue depth indicates a different problem — memory pressure, a slow downstream dependency, degraded hardware.

GPU utilization alongside memory bandwidth utilization. As noted above, compute utilization alone is an incomplete picture; tracking memory bandwidth utilization and KV cache occupancy distinguishes compute-bound from memory-bound bottlenecks, which call for different fixes — batching changes address the former, KV cache management and paging address the latter.

Error and rejection rates, broken out by cause. Timeouts, out-of-memory errors, and admission-control rejections are different failure modes with different remediations; aggregating them into a single “error rate” obscures which one is actually driving reliability problems.

Building this observability layer is not optional overhead — it is the primary tool for distinguishing a system that needs more hardware from one that needs a different batching strategy.

Observability at the serving layer answers “is the system performing well,” which is a different question from “does the model perform well” — the domain of evaluation methodology rather than infrastructure metrics. A serving system can hit every latency and throughput target while serving a model whose outputs have degraded; neither layer of monitoring substitutes for the other. Understanding what a deployed model is doing internally, the subject of neural network interpretability methods, is likewise a separate concern from serving performance, though the two increasingly intersect as teams instrument production models for both latency and behavioral drift.

Frequently Asked Questions

What is the difference between static and continuous batching?

Static batching groups a fixed set of requests and holds them together until the whole batch finishes, so short requests wait behind long ones. Continuous batching operates at the level of individual decoding steps: finished sequences are evicted immediately and replaced with new requests from the queue, keeping GPU occupancy high without short requests waiting on longer ones.

How does PagedAttention improve LLM serving throughput?

PagedAttention allocates each sequence’s key-value cache in fixed-size, non-contiguous memory blocks rather than one worst-case-sized region per sequence, borrowing the paging concept from operating system virtual memory. This cuts memory fragmentation and lets a serving system fit more concurrent sequences into GPU memory, which is what lets continuous batching reach substantially higher throughput than naive request-level batching.

Why is autoscaling harder for large models than for typical web services?

Loading a large model’s weights onto a GPU and warming up compiled kernels can take seconds to over a minute, unlike the near-instant startup of a typical web container. This cold-start cost means reactive autoscaling arrives too late for traffic spikes, pushing teams toward warm pools, predictive scaling, or leading-indicator triggers like queue depth rather than GPU utilization alone.

What is LoRA adapter swapping and why does it matter for multi-model serving?

LoRA (Low-Rank Adaptation) fine-tunes a small set of extra parameters atop a frozen base model rather than duplicating the full model per task. Because adapters are small, a serving system can keep one base model resident on a GPU and swap adapters per request, letting many fine-tunes share a single deployment instead of needing one per variant.

Which serving framework should a team choose?

There is no universally correct choice — it depends on model diversity, existing infrastructure, and traffic patterns. Framework-agnostic servers suit heterogeneous portfolios; LLM-specialized engines suit teams running mostly autoregressive generation at scale; framework-native tools suit teams standardized on one training framework; Kubernetes-native orchestration suits teams needing standardized deployment and autoscaling across engines. Many fleets combine more than one category.