Bayesian Optimization: A quick note

In recent years the need for Automated “ Auto-ML” approaches has gained lot of interest, mainly to move away from hand-tuning of the hyperparameter to the automatic tuning of these parameters or select machine learning models. Some of the examples relate optimization/tuning various parameters of the deep neural network e.g. learning rate, number of parameters/layers, batch size etc deep neural networks, ensembled models, etc etc.

With a large range of the parameter, random choices of these parameter via hand tuning or approaches e.g. grid search can be extremely time-consuming and or costly. This is specially if the training times are to two-three weeks on TPU/GPU cluster.

Bayesian optimization is a key tool in “automatic” tuning of the parameters. The sequential process of BO is illustrated. BO employs a probabilistic model e.g. Gaussian process over in the initially unknown objective function g. For every iteration t, prior i.e. GP(g), is evaluated to obtain update the GP(y*) i.e. posterior, to estimate the quality of improving with that set of parameters.

The evaluation is performed by the mean of “acquisition function” and this function permits assessment and selection of the next set of parameters to evaluate for next iteration such as to obtain maximal improvement. This process fundamentally translates in the uncertainty…