Every machine learning model has two categories of numbers attached to it. The first category — weights, embeddings, biases — is learned directly from data through gradient descent. The second category is set before training even begins, and no amount of gradient descent will find good values for it on its own. Learning rate, batch size, the number of layers, dropout probability, the regularization coefficient — these are hyperparameters, and choosing them badly can waste a training run as thoroughly as a bug in the model code.

The distinction matters because the two categories require entirely different search strategies. Parameters are optimized by differentiating a loss function with respect to them. Hyperparameters are optimized by treating the entire training-and-evaluation process as a black-box function: pick a configuration, train a model, measure validation performance, repeat. That outer loop is expensive — each evaluation can cost hours or days of compute — which is precisely why the method used to search hyperparameter space matters as much as the search space itself.

This article covers the four approaches practitioners actually reach for: grid search, random search, Bayesian optimization, and population-based training (PBT). Each has a different answer to the same underlying question — given a fixed compute budget, how do you spend it to find the best configuration?

Parameters vs. Hyperparameters: Why the Distinction Matters

A learned parameter — a weight in a linear layer, a convolution kernel value, an attention projection matrix — is optimized by the training algorithm itself. You initialize it, compute a loss, backpropagate a gradient, and update it. The process is automatic and differentiable end to end.

A hyperparameter sits outside that loop. It controls how the training algorithm behaves rather than being shaped by it. Learning rate determines the step size of every gradient update. Batch size determines how many examples are averaged before each update, which affects both the noise in the gradient estimate and the achievable parallelism. Network depth and width determine model capacity. Weight decay and dropout rate control regularization strength. None of these can be learned by gradient descent on the training loss, because the training loss isn’t differentiable with respect to them in any useful sense — changing the learning rate doesn’t produce a gradient, it produces an entirely different trajectory through parameter space.

This is why hyperparameter tuning is treated as a separate, outer optimization problem: search over configurations, where evaluating a single configuration means running the inner optimization (full model training) to convergence or some proxy for convergence. The objective function is expensive to evaluate, noisy (different random seeds give different results for the same configuration), and typically has no closed-form gradient. That combination of properties — expensive, noisy, gradient-free — is what shapes every method described below.

The most direct approach to hyperparameter search is grid search: define a discrete set of values for each hyperparameter, form the Cartesian product across all hyperparameters, and evaluate every resulting configuration. For two hyperparameters with 5 values each, that’s 25 configurations. For five hyperparameters with 5 values each, it’s 3,125. The combinatorial explosion is grid search’s defining weakness — the number of evaluations required grows exponentially with the number of hyperparameters being tuned.

Grid search also wastes evaluations in a specific, provable way when hyperparameters vary in how much they affect the outcome. Bergstra and Bengio (2012) formalized this in “Random Search for Hyper-Parameter Optimization,” showing that if only a few hyperparameters actually matter for a given problem — which is empirically the common case — grid search allocates its evaluation budget inefficiently. Consider two hyperparameters, one that strongly affects validation loss and one that barely matters. A grid with 5 values per dimension tests only 5 distinct values of the important hyperparameter, because every value of the unimportant one is paired with the same 5 values of the important one. Random search, by contrast, draws each configuration independently from a distribution over the hyperparameter space. With the same 25-evaluation budget, random search tests 25 distinct values of the important hyperparameter, because no two draws share the same value along any single axis. The paper reports that random search found models matching or exceeding grid search’s best results using a small fraction of the compute, across several neural network configuration tasks — a result that has since been reproduced widely enough that random search is now the default baseline against which more sophisticated methods are judged, not grid search.

The practical rule that follows: grid search should be treated as ill-advised whenever more than two or three hyperparameters are being tuned simultaneously, and random search should be the default even for smaller search spaces, because it strictly dominates grid search’s coverage properties per evaluation. The remaining question is whether it’s worth doing better than random search — and for that, you need a method that uses the results of past evaluations to choose the next one, rather than sampling blind.

Bayesian Optimization

Random search treats every evaluation as independent — it learns nothing from the ten configurations it has already tried when choosing the eleventh. Bayesian optimization is designed specifically to close that gap. It builds a probabilistic model of the objective function — validation loss as a function of hyperparameters — based on the evaluations performed so far, and uses that model to decide which configuration to try next.

The probabilistic model is called a surrogate model, and the most common choice is a Gaussian process (GP). A Gaussian process defines a distribution over functions: given the configurations evaluated so far and their observed validation scores, it produces a posterior mean (the model’s best guess of validation loss at any untested configuration) and a posterior variance (how uncertain that guess is). Regions far from any tested configuration have high variance; regions near several tested configurations with similar scores have low variance. Snoek, Larochelle, and Adams (2012) demonstrated this approach applied directly to tuning machine learning models in “Practical Bayesian Optimization of Machine Learning Algorithms,” including handling of the practical complications — variable evaluation cost, parallel evaluation, and problem-specific priors — that a textbook GP treatment glosses over.

The surrogate model alone doesn’t tell you where to sample next; that decision is made by an acquisition function, which converts the posterior mean and variance into a single score representing the expected value of evaluating a given configuration. Expected improvement (EI) is the most widely used acquisition function: it computes the expected amount by which a candidate configuration would improve on the best result seen so far, weighting both the chance of improvement (favoring low posterior mean, in a minimization setting) and the magnitude of possible improvement (favoring high posterior variance — unexplored regions might hide a much better result). Upper confidence bound (UCB) and probability of improvement are the other common acquisition functions; each trades off exploitation (sampling near known good configurations) against exploration (sampling in uncertain regions) slightly differently.

The practical benefit is sample efficiency: Bayesian optimization typically finds strong configurations in far fewer evaluations than random search, which matters when each evaluation is a multi-hour or multi-day training run. The cost is that the surrogate model itself becomes expensive to fit and query as the number of observations grows — standard GP inference scales cubically with the number of observations — and Bayesian optimization does not parallelize as naturally as random search, since each new configuration ideally depends on the results of the ones before it. Tree-structured Parzen estimators (TPE), used in libraries like Hyperopt and Optuna, sidestep some of the GP scaling issues by modeling the distribution of good and bad configurations separately rather than fitting a single joint surrogate.

Population-Based Training

Every method described so far treats hyperparameters as fixed for the duration of a training run: you pick a learning rate, train to convergence, and evaluate. Population-based training, introduced by Jaderberg et al. (2017) at DeepMind, discards that assumption. Instead of searching for a single static configuration, PBT searches for a schedule — hyperparameter values that change over the course of training.

The mechanism works by training a population of models in parallel, each with its own hyperparameter configuration, and periodically comparing them. When one member of the population is performing poorly relative to the others, it is replaced: its weights are overwritten with a copy of a better-performing member’s weights (this step is called exploit), and its hyperparameters are then perturbed — typically scaled up or down by a random factor, or resampled from a defined range (this step is called explore). Training then continues from the copied weights under the new hyperparameters. Over many rounds of exploit-and-explore, the population converges toward both good weights and a hyperparameter schedule that adapts to the current state of training — for instance, a higher learning rate early in training and a lower one as the model approaches convergence, discovered automatically rather than specified by a hand-tuned decay schedule.

The original paper reported gains across three quite different domains: deep reinforcement learning, machine translation (measured by BLEU score), and GAN training (measured by Inception score), with PBT finding configurations that outperformed both hand-tuned baselines and standard hyperparameter search methods at comparable wall-clock cost. The key practical distinction from Bayesian optimization is that PBT never discards a partially-trained model — the “exploit” step transfers weights, not just hyperparameter values, so compute already spent training a model isn’t thrown away just because its current configuration was weak. That makes PBT particularly well suited to problems (like RL and GAN training) where the optimal hyperparameters genuinely change over the course of training and where evaluation is a single continuous process rather than a series of independent short runs.

Choosing a Method: Compute Budget and When the Overhead Pays Off

None of the four methods is universally correct; each fits a different compute regime and problem shape.

Random search should be the default whenever the search space is small (fewer than roughly five to ten hyperparameters), each individual training run is cheap enough to run dozens or hundreds of times, and evaluations are easy to parallelize across independent machines with no coordination needed. It requires no additional infrastructure beyond a random sampler and a job queue, and per the Bergstra and Bengio result, it will not waste evaluations the way a grid does.

Bayesian optimization earns its overhead when individual evaluations are expensive enough that sample efficiency dominates — large models where a single training run costs meaningful GPU-hours, and where you can only afford a few dozen total evaluations rather than a few hundred. The infrastructure cost is real: fitting and querying a surrogate model, choosing and tuning an acquisition function, and accepting reduced parallelism relative to random search. Below roughly ten to twenty evaluations the surrogate model has too little data to add value over random sampling; above a few hundred hyperparameters, GP-based approaches specifically start to struggle with dimensionality and inference cost, favoring TPE-based alternatives instead.

Population-based training is worth its substantially higher infrastructure complexity — a population of models training in parallel, a scheduler that periodically ranks and copies weights, and checkpointing infrastructure to support the exploit step — specifically for problems where hyperparameters are known or suspected to need to change over training, and where the training process is long-running and continuous rather than a sequence of independent short jobs. Reinforcement learning, GAN training, and other unstable or non-stationary optimization processes are the strongest fits; a straightforward supervised classification task with a well-understood learning rate schedule usually doesn’t need it.

A reasonable default sequence: start with random search to establish a baseline and sanity-check the search space, move to Bayesian optimization (via a library like Optuna) once evaluation cost justifies the added sample efficiency, and reserve PBT for cases — RL, adversarial training, long training runs — where a static configuration is a real bottleneck.

It’s also worth noting what none of these methods solve: they optimize a validation metric you specify, on a search space you define. A poorly chosen search space or a validation metric that doesn’t reflect the deployment objective produces a well-optimized answer to the wrong question, regardless of which search method is used. These methods are complementary to techniques like parameter-efficient fine-tuning with LoRA and QLoRA, which reduce the cost of each training run and change the calculus of how many evaluations are affordable in the first place. Similarly, systems that combine a fine-tuned or prompted model with external context — as covered in our piece on retrieval-augmented generation in practice — introduce their own hyperparameters (retrieval depth, chunk size, reranking thresholds) frequently tuned with the same methods described here, applied to end-to-end task metrics rather than training loss alone.

Frequently Asked Questions

What’s the difference between a parameter and a hyperparameter?

A parameter (a weight, bias, or embedding value) is learned automatically by gradient descent during training. A hyperparameter (learning rate, batch size, number of layers, regularization strength) is set before training starts and controls how the training process itself behaves. Hyperparameters can’t be learned by backpropagation because changing them doesn’t produce a usable gradient with respect to the training loss.

Grid search allocates its evaluation budget evenly across every hyperparameter combination, which wastes evaluations when only a subset of hyperparameters actually affect performance. Random search samples each configuration independently, so with the same budget it tests more distinct values along each important dimension. Bergstra and Bengio (2012) showed this empirically across multiple neural network tasks.

When is Bayesian optimization worth the added complexity?

Bayesian optimization pays off when each training run is expensive enough — hours to days of compute — that you can only afford a limited number of evaluations, typically a few dozen. Its surrogate model uses prior evaluation results to choose promising configurations rather than sampling blindly, improving sample efficiency at the cost of reduced parallelism and additional infrastructure to fit and query the model.

What makes population-based training different from other hyperparameter search methods?

PBT tunes a schedule of hyperparameter values that change during training, rather than a single static configuration. It trains a population of models in parallel, periodically replacing weak performers with copies of strong performers’ weights, then perturbing their hyperparameters. This preserves partially trained weights instead of discarding them, making it well suited to reinforcement learning and other training processes where optimal hyperparameters shift over time.

Do larger models need more hyperparameter tuning or less?

Larger models often need less exhaustive tuning per hyperparameter dimension because empirical scaling studies have established reasonably reliable defaults and scaling rules for learning rate and batch size as model size grows. But because each evaluation is far more expensive at scale, teams typically shift from random search toward Bayesian optimization or smaller, more targeted search spaces to conserve compute rather than tuning less carefully.