4.13 SMACdown

SMAC is a meta-algorithm for the optimization of any candidate function against a set of possible parameter values. For example, the function may be a random forest learning algorithm. In that case, the parameters would be the number of trees, number of random candidates, pruning or no pruning, etc. The criterion could be the error of the generated model on a holdout set.

WhizzML provides an implementation of the SMAC algorithm, dumbed down so that one can insert it somewhere in an existing optimization pipeline regardless of the context.

To use the provided function, smacdown-optimize one need only provide two functions as arguments, a generator and an evaluator. The generator takes no arguments and returns a random set of parameters to be evaluated as a map. For example:

{"number_of_trees" 57
   "random_candidates" 20
   "pruning" true}

The values of the parameters can be numbers, strings, or boolean values. There are no further restrictions on the form of the map, except that every generated map must have the same set of keys.

The evaluator takes a list of parameter maps, of the sort generated by generator, evaluates these candidates using the chosen criterion, and returns the resulting list of evaluations. For example, the evaluator may train a model using each set of candidate parameters provided, then evaluate those models on a holdout set of data, returning the error of each one. Note that the criterion must be constructed so that lower values indicate higher quality parameters. Said another way, SMACdown will search for the set of parameters that minimizes the objective criterion.

Given these two functions, smacdown-optimize learns a clever search strategy for the best set of parameters. It returns a list of all of the maps of parameters it has evaluated, sorted high to low by quality, so the lowest score for the objective criterion is first. The objective criterion for each of the output maps is stored under the key actual_value and the parameters evaluated are stored under the key parameters.