Optimization with continuous/integer parameters and constraints

Hello

I'm using different classifiers to solve a classification problem which consists of two classes. Of course I have to tune the hyperparameters which I want to treat as an optimization problem. The cost function is the (cross-validated) accuracy. I have the following three different optimization problems:

1. Continuous parameters + 1-2 integer valued parameters to optimize (perhaps the integer-valued parameters could be dropped and optimized separately) Bound constraints on the parameters (lower bound and upper bounds) Equality constraint on some parameters => sum(w(i)) = 1 for weight parameters w(i) where 0 <= w(i) <= 1. The weights are used in a weighted RBF kernel for SVM.

2. Continuous parameters only Bound constraints on the parameters

3. Integer valued parameters only Bound constraints on the parameters

I have started with Nelder-Mead using the Matlab's fminsearch function but this seems not to be optimal for 1. and 3. Which optimization procedures would you recommend to use for these three types? One possibility is mixed integer programming using the ga function of Matlab but according to the documentation the equality constraints could be a problem.

Additionaly, I have a distance metric parameter which is a string (e.g. euclidean distance etc.). Should I treat this as an integer in the optimization procedure? For example, mapping euclidean to 1, mahalanobis to 2 etc.

On top of that, an additional problem is the local minima. How could this be solved efficiently (without using random restarts which is a bit time consuming)?

Answer 1

[Don Mathis]

The bayesopt function in the Statistics & Machine Learning Toolbox release R2016b can handle all of these problems.

The tricky one is the constraint in which sum(w(i)) = 1. With N weights you could define N parameters p(i) to be optimized in the range, say [-10,10], and then inside your objective function set w(i) = exp(p(i))/sum_j(exp(p(j))