Purpose

This Vignette is supposed to give you an introduction on how to use mlrMBO for hyperparameter tuning in the context of machine learning using the mlr package.

mlr

For the purpose of hyperparameter tuning, we will use the mlr package. mlr provides a framework for machine learning in R that comes with a broad range of machine learning functionalities and is easily extendable. One possible approach is to use mlr to train a learner and evaluate its performance for a given hyperparameter configuration in the objective function. Alternatively, we can access mlrMBO’s model-based optimization directly using mlr’s tuning functionalities. This yields the benefit of integrating hyperparameter tuning with model-based optimization into your machine learning experiments without any overhead.

Preparations

First, we load the required packages. Next, we configure mlr to suppress the learner output to improve output readability. Additionally, we define a global variable giving the number of tuning iterations. Note that this number is set (very) low to reduce runtime.

library(mlrMBO)
library(mlr)

configureMlr(on.learner.warning = "quiet", show.learner.output = FALSE)

iters = 5

1 Custom objective function to evaluate performance

As an example, we tune the cost and the gamma parameter of a rbf-SVM on the Iris data. First, we define the parameter set. Note that the transformations added in the trafo argument mean, that we tune the parameters on a logarithmic scale.

par.set = makeParamSet(
  makeNumericParam("cost", -15, 15, trafo = function(x) 2^x),
  makeNumericParam("gamma", -15, 15, trafo = function(x) 2^x)
)

Next, we define the objective function. First, we define a learner and set its hyperparameters by using makeLearner. To evaluate its performance we use the resample function which automatically takes care of fitting the model and evaluating it on a test set. In this example, resampling is done using 3-fold cross-validation, by passing the ResampleDesc object cv3, that comes predefined with mlr, as an argument to resample. The measure to be optimized can be specified (e.g by passing measures = ber, for the balanced error rate), however mlr has a default for each task type. For classification the mmce(Mean misclassification rate) is the default. Like in this example, we set minimize = TRUE and has.simple.signature = FALSE. Note that the iris.task is provided automatically when loading mlr.

svm = makeSingleObjectiveFunction(name = "svm.tuning",
  fn = function(x) {
    lrn = makeLearner("classif.svm", par.vals = x)
    resample(lrn, iris.task, cv3, show.info = FALSE)$aggr
  },
  par.set = par.set,
  noisy = TRUE,
  has.simple.signature = FALSE,
  minimize = TRUE
)

Now we create a default MBOControl object and tune the rbf-SVM.

ctrl = makeMBOControl()
ctrl = setMBOControlTermination(ctrl, iters = iters)

res = mbo(svm, control = ctrl, show.info = FALSE)
print(res)
## Recommended parameters:
## cost=12.5; gamma=-11.3
## Objective: y = 0.033
## 
## Optimization path
## 8 + 5 entries in total, displaying last 10 (or less):
##          cost      gamma          y dob eol error.message exec.time
## 4  -11.945100  -2.068064 0.66666667   0  NA          <NA>     0.077
## 5    3.037741  -4.453539 0.04666667   0  NA          <NA>     0.074
## 6   14.228784   5.399369 0.55333333   0  NA          <NA>     0.079
## 7   10.329282 -12.640421 0.04000000   0  NA          <NA>     0.075
## 8   -4.836816   7.502343 0.70666667   0  NA          <NA>     0.079
## 9    3.036409  -3.700924 0.05333333   1  NA          <NA>     0.248
## 10   3.220775 -14.985505 0.72666667   2  NA          <NA>     0.072
## 11   0.338691  -3.202265 0.03333333   3  NA          <NA>     0.070
## 12  10.134825  -8.557274 0.05333333   4  NA          <NA>     0.072
## 13  12.455868 -11.304792 0.03333333   5  NA          <NA>     0.075
##             cb error.model train.time  prop.type propose.time        se
## 4           NA        <NA>         NA initdesign           NA        NA
## 5           NA        <NA>         NA initdesign           NA        NA
## 6           NA        <NA>         NA initdesign           NA        NA
## 7           NA        <NA>         NA initdesign           NA        NA
## 8           NA        <NA>         NA initdesign           NA        NA
## 9  -0.01696749        <NA>      0.243  infill_cb        0.176 0.2594734
## 10 -0.10054011        <NA>      0.592  infill_cb        0.072 0.1626048
## 11 -0.12261760        <NA>      0.042  infill_cb        0.067 0.1995591
## 12 -0.10828749        <NA>      0.040  infill_cb        0.069 0.1957591
## 13 -0.07632605        <NA>      0.043  infill_cb        0.074 0.1475221
##          mean lambda
## 4          NA     NA
## 5          NA     NA
## 6          NA     NA
## 7          NA     NA
## 8          NA     NA
## 9  0.24250588      1
## 10 0.06206465      1
## 11 0.07694148      1
## 12 0.08747158      1
## 13 0.07119607      1
res$x
## $cost
## [1] 12.45587
## 
## $gamma
## [1] -11.30479
res$y
## [1] 0.03333333
op = as.data.frame(res$opt.path)
plot(cummin(op$y), type = "l", ylab = "mmce", xlab = "iteration")

2 Using mlr’s tuning interface

Instead of defining an objective function where the learner’s performance is evaluated, we can make use of model-based optimization directly from mlr. We just create a TuneControl object, passing the MBOControl object to it. Then we call tuneParams to tune the hyperparameters.

ctrl = makeMBOControl()
ctrl = setMBOControlTermination(ctrl, iters = iters)
tune.ctrl = makeTuneControlMBO(mbo.control = ctrl)
res = tuneParams(makeLearner("classif.svm"), iris.task, cv3, par.set = par.set, control = tune.ctrl,
  show.info = FALSE)
print(res)
## Tune result:
## Op. pars: cost=1.62e+04; gamma=0.000177
## mmce.test.mean=0.0267
res$x
## $cost
## [1] 16165.61
## 
## $gamma
## [1] 0.0001771411
res$y
## mmce.test.mean 
##     0.02666667
op.y = getOptPathY(res$opt.path)
plot(cummin(op.y), type = "l", ylab = "mmce", xlab = "iteration")

Hierarchical mixed space optimization

In many cases, the hyperparameter space is not just numerical but mixed and often even hierarchical. This can easily be done out-of-the-box and needs no adaption to our previous example. (Recall that a suitable surrogate model is chosen automatically, as explained here.) To demonstrate this, we tune the cost and the kernel parameter of a SVM. When kernel takes the radial value, gamma needs to be specified. For a polynomial kernel, the degree needs to be specified.

par.set = makeParamSet(
  makeDiscreteParam("kernel", values = c("radial", "polynomial", "linear")),
  makeNumericParam("cost", -15, 15, trafo = function(x) 2^x),
  makeNumericParam("gamma", -15, 15, trafo = function(x) 2^x, requires = quote(kernel == "radial")),
  makeIntegerParam("degree", lower = 1, upper = 4, requires = quote(kernel == "polynomial"))
)

Now we can just repeat the setup from the previous example and tune the hyperparameters.

ctrl = makeMBOControl()
ctrl = setMBOControlTermination(ctrl, iters = iters)
tune.ctrl = makeTuneControlMBO(mbo.control = ctrl)
res = tuneParams(makeLearner("classif.svm"), iris.task, cv3, par.set = par.set, control = tune.ctrl,
  show.info = FALSE)

Parallelization and multi-point proposals

We can easily add multi-point proposals and parallelize it using the parallelMap package. (Note that the chosen multicore back-end for parallelization does not work on windows machines. Please refer to the parallelization section for details on parallelization and multi point proposals.) In each iteration, we propose as many points as CPUs used for parallelization. As infill criterion we use Expected Improvement.

library(parallelMap)
ncpus = 2L

ctrl = makeMBOControl(propose.points = ncpus)
ctrl = setMBOControlTermination(ctrl, iters = iters)
ctrl = setMBOControlInfill(ctrl, crit = crit.ei)
ctrl = setMBOControlMultiPoint(ctrl, method = "cl", cl.lie = min)
tune.ctrl = makeTuneControlMBO(mbo.control = ctrl)
parallelStartMulticore(cpus = ncpus)
## Starting parallelization in mode=multicore with cpus=2.
res = tuneParams(makeLearner("classif.svm"), iris.task, cv3, par.set = par.set, control = tune.ctrl, show.info = FALSE)
## Mapping in parallel: mode = multicore; cpus = 2; elements = 16.
## Mapping in parallel: mode = multicore; cpus = 2; elements = 2.
## Mapping in parallel: mode = multicore; cpus = 2; elements = 2.
## Mapping in parallel: mode = multicore; cpus = 2; elements = 2.
## Mapping in parallel: mode = multicore; cpus = 2; elements = 2.
## Mapping in parallel: mode = multicore; cpus = 2; elements = 2.
parallelStop()
## Stopped parallelization. All cleaned up.

Usecase: Pipeline configuration

It is also possible to tune a whole machine learning pipeline, i.e. preprocessing and model configuration. The example pipeline is: * Feature filtering based on an ANOVA test or covariance, such that between 50% and 100% of the features remain. * Select either a SVM or a naive Bayes classifier. * Tune parameters of the selected classifier.

First, we define the parameter space:

par.set = makeParamSet(
  makeDiscreteParam("fw.method", values = c("anova.test", "variance")),
  makeNumericParam("fw.perc", lower = 0.1, upper = 1),
  makeDiscreteParam("selected.learner", values = c("classif.svm", "classif.naiveBayes")),
  makeNumericParam("classif.svm.cost", -15, 15, trafo = function(x) 2^x,
    require = quote(selected.learner == "classif.svm")),
  makeNumericParam("classif.svm.gamma", -15, 15, trafo = function(x) 2^x,
    requires = quote(classif.svm.kernel == "radial" & selected.learner == "classif.svm")),
  makeIntegerParam("classif.svm.degree", lower = 1, upper = 4,
    requires = quote(classif.svm.kernel == "polynomial" & selected.learner == "classif.svm")),
  makeDiscreteParam("classif.svm.kernel", values = c("radial", "polynomial", "linear"),
    require = quote(selected.learner == "classif.svm"))
)

Next, we create the control objects and a suitable learner, combining makeFilterWrapper() with makeModelMultiplexer(). (Please refer to the advanced tuning chapter of the mlr tutorial for details.) Afterwards, we can run tuneParams() and check the results.

ctrl = makeMBOControl()
ctrl = setMBOControlTermination(ctrl, iters = iters)
lrn = makeFilterWrapper(makeModelMultiplexer(list("classif.svm", "classif.naiveBayes")))
tune.ctrl = makeTuneControlMBO(mbo.control = ctrl)

res = tuneParams(lrn, iris.task, cv3, par.set = par.set, control = tune.ctrl, show.info = FALSE)
print(res)
## Tune result:
## Op. pars: fw.method=variance; fw.perc=0.75; selected.learner=classif.naiv...
## mmce.test.mean=0.0333
res$x
## $fw.method
## [1] "variance"
## 
## $fw.perc
## [1] 0.7496524
## 
## $selected.learner
## [1] "classif.naiveBayes"
res$y
## mmce.test.mean 
##     0.03333333
op = as.data.frame(res$opt.path)
plot(cummin(op$mmce.test.mean), type = "l", ylab = "mmce", xlab = "iteration")