{"id":104,"date":"2026-09-24T14:42:38","date_gmt":"2026-09-24T14:42:38","guid":{"rendered":"https:\/\/evoxia.online\/?p=104"},"modified":"2026-09-24T14:42:38","modified_gmt":"2026-09-24T14:42:38","slug":"bayesian-optimization-and-hyperparameter-tuning-making-every-expensive-experiment-count","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=104","title":{"rendered":"Bayesian Optimization and Hyperparameter Tuning: Making Every Expensive Experiment Count"},"content":{"rendered":"\n

The Hyperparameter Problem<\/strong><\/p>\n\n\n\n

Every machine learning model comes with settings that training does not learn on its own: the learning rate, the number of layers, the batch size, the amount of regularization, the depth of each tree in a boosted ensemble. These are hyperparameters, and they can make the difference between a model that works and one that does not.<\/p>\n\n\n\n

Finding good values is an optimization problem with an unpleasant objective function. Each evaluation means training a model and measuring its validation performance, which can take minutes, hours or days. There is no gradient available. The result is noisy, because random initialization and data shuffling change it from run to run. And some hyperparameters are continuous, some are integers and some are categorical choices. The question is how to find a good configuration while spending as few training runs as possible.<\/p>\n\n\n\n

Grid Search and Random Search<\/strong><\/p>\n\n\n\n

The traditional approach was grid search: choose a few values for each hyperparameter and try every combination. It is simple and easy to parallelize, but the number of combinations grows exponentially with the number of hyperparameters, and it wastes most of its budget.<\/p>\n\n\n\n

James Bergstra and Yoshua Bengio showed why in a 2012 paper in the Journal of Machine Learning Research. In most problems, only a few hyperparameters really matter, and which ones matter depends on the dataset. A grid with 9 points over two hyperparameters tests only 3 distinct values of each. Nine random points test 9 distinct values of each. If only one of the two hyperparameters matters, random search explores it three times more thoroughly for the same cost. Their experiments found that random search matched or beat grid search on neural networks with a fraction of the budget. Random search has been the standard baseline since then.<\/p>\n\n\n\n

Both methods share a weakness: they ignore what they have already learned. The fiftieth trial is chosen with no knowledge of the first forty-nine.<\/p>\n\n\n\n

The Idea Behind Bayesian Optimization<\/strong><\/p>\n\n\n\n

Bayesian optimization uses the results so far to decide where to look next. It has two components. The first is a surrogate model, a cheap probabilistic model of the expensive objective, fitted to all the evaluations made so far. The second is an acquisition function, which uses the surrogate to score every candidate point by how useful it would be to evaluate there.<\/p>\n\n\n\n

The loop is: fit the surrogate to the data, find the point that maximizes the acquisition function (cheap, since it only involves the surrogate), evaluate the real objective at that point (expensive), add the result to the data, and repeat. Much more computation goes into choosing each point than random search spends, but that is a good trade when each real evaluation costs hours of GPU time.<\/p>\n\n\n\n

The ideas go back to Jonas Mockus in the 1970s. Jones, Schonlau and Welch made it practical for engineering in 1998 with their Efficient Global Optimization (EGO) algorithm. Jasper Snoek, Hugo Larochelle and Ryan Adams brought it to machine learning in their 2012 NeurIPS paper \u00abPractical Bayesian Optimization of Machine Learning Algorithms,\u00bb which showed it finding better hyperparameters than human experts on several models, including convolutional networks.<\/p>\n\n\n\n

Gaussian Processes as Surrogates<\/strong><\/p>\n\n\n\n

The classic surrogate is a Gaussian process (GP). A GP defines a distribution over functions. After seeing some evaluations, it gives, for any untested point, both a predicted value (the mean) and an uncertainty (the variance). Near points already evaluated, the uncertainty is small. Far from them, it is large.<\/p>\n\n\n\n

How the GP generalizes between points is set by its kernel, which encodes assumptions such as how smooth the objective is. The Mat\u00e9rn 5\/2 kernel, which Snoek and colleagues recommended, assumes a function that is smooth but not unrealistically so. The kernel’s own parameters, such as length scales, are typically fitted to the data or averaged over.<\/p>\n\n\n\n

GPs have limits. Their cost grows cubically with the number of observations, which is fine for a few hundred evaluations but not for tens of thousands. They also struggle with more than roughly 20 dimensions and with categorical or conditional hyperparameters (the number of units in layer three only matters if there is a layer three).<\/p>\n\n\n\n

Acquisition Functions: Exploration Versus Exploitation<\/strong><\/p>\n\n\n\n

The acquisition function decides between exploitation (evaluating near the best result found so far, where the surrogate predicts good values) and exploration (evaluating where the surrogate is uncertain, since something better might be hiding there).<\/p>\n\n\n\n

Expected improvement (EI) is the most common. For each candidate point, it computes how much the result is expected to beat the current best, averaging over the surrogate’s uncertainty. A point with a slightly worse predicted mean but high uncertainty can have a higher EI than a point with a good mean and no uncertainty. Probability of improvement is a simpler version that ignores how large the improvement would be. The upper confidence bound (UCB) adds a multiple of the standard deviation to the mean and picks the highest. Srinivas and colleagues proved in 2010 that GP-UCB, with an appropriate schedule for that multiple, has sublinear regret, a formal guarantee that it keeps getting closer to the optimum.<\/p>\n\n\n\n

Tree-Structured Parzen Estimators and Other Surrogates<\/strong><\/p>\n\n\n\n

Bergstra and colleagues introduced another approach in 2011, the Tree-structured Parzen Estimator (TPE). Instead of modeling the objective directly, TPE splits the observations into a good group and a bad group based on a quantile threshold and models the distribution of hyperparameter values within each group. It then samples candidates that are likely under the good distribution and unlikely under the bad one. TPE handles conditional and categorical spaces naturally and scales well, so it became the default algorithm in Hyperopt and later in Optuna, the framework from Akiba and colleagues presented at KDD 2019. SMAC, from Hutter, Hoos and Leyton-Brown, uses random forests as the surrogate for similar reasons.<\/p>\n\n\n\n

Stopping Bad Runs Early<\/strong><\/p>\n\n\n\n

A different way to save compute is to not finish training for configurations that are clearly failing. Successive halving starts many configurations with a small budget (a few epochs), keeps the best half, doubles their budget, and repeats. Hyperband, from Li and colleagues (2017), runs successive halving at several trade-offs between the number of configurations and the budget per configuration, since it is not known in advance how early a good run can be recognized.<\/p>\n\n\n\n

BOHB, from Falkner, Klein and Hutter (2018), combined the two ideas: Hyperband decides how much budget each run gets, and a TPE-style model picks which configurations to try. Optuna implements the same principle through pruners, which stop trials whose intermediate validation scores fall below those of earlier trials at the same step.<\/p>\n\n\n\n

Tuning AlphaGo<\/strong><\/p>\n\n\n\n

A concrete example comes from DeepMind. In a 2018 paper, Yutian Chen and colleagues described how they used Bayesian optimization to tune AlphaGo’s hyperparameters, including those of its Monte Carlo tree search, before the 2016 match against Lee Sedol. Each evaluation meant playing many self-play games. According to the paper, tuning raised the win rate against the untuned version from 50% to 66.5% in self-play, and the tuned version was the one used in the final match.<\/p>\n\n\n\n

Beyond hyperparameters, the same method is used wherever experiments are expensive: tuning chemical reaction conditions in self-driving laboratories, designing materials, calibrating simulators, placing sensors, and A\/B testing configurations of web services.<\/p>\n\n\n\n

Limits and Practical Advice<\/strong><\/p>\n\n\n\n

Bayesian optimization is most valuable when evaluations are expensive, the number of hyperparameters is moderate, and the budget is small (tens to hundreds of trials). With cheap evaluations and large parallel resources, random search with early stopping is often just as good and much simpler. In high dimensions, most of the surrogate’s effort goes into regions that do not matter.<\/p>\n\n\n\n

It also cannot fix a badly designed search space. If the learning rate range does not contain a good value, no algorithm will find one. Searching over the logarithm of scale parameters like learning rate and regularization strength, and setting ranges based on experience, often matters more than the choice of optimizer.<\/p>\n\n\n\n

Reflection<\/strong><\/p>\n\n\n\n

Bayesian optimization makes sense because each training run is expensive. When evaluations are cheap, trying many random points works fine. When each one costs a day of GPU time, it is worth spending a lot of computation on deciding what to try next, and a surrogate that tracks both its predictions and its uncertainty is a principled way to do that. The same reasoning is spreading beyond machine learning to laboratories and factories, where each experiment costs real materials and time.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

The Hyperparameter Problem Every machine learning model comes with settings that training does not learn on its own: the learning rate, the number of layers, the batch size, the amount of regularization, the depth of each tree in a boosted ensemble. These are hyperparameters, and they can make the difference between a model that works […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-104","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/104","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=104"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/104\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=104"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=104"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=104"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}