Question of Hyperparameter tuning of shallow neural network

Hi all,

I trained a shallow neuron network and tuning parameters following this post from @Greg Heath.

https://ch.mathworks.com/matlabcentral/answers/267380-the-best-ann-configuration?s_tid=srchtitle

I only tuning the number of nuerons in one hidden layer and accept all default parameters.

here is comments from referee：

The machine learning models have numerous hyper-parameters, but the authors only optimise one, and did not even report the values of the others. It is not clear why they chosen only one, or why they chose the ones they did, but this is unacceptable. All independent hyper-parameters need to be optimised simultaneously.

how to answer this comment?

Thank you.

Answer 1

[Shashank Gupta]

Hi Qiang,

The points raised by the reviewers are valid, you can't just optimize one variable and leave the rest as it is. even if you are intent to optimize some of the variable, you must explain why it is done and must provide the values used for rest of the parameters. All the hyper parameters used in ML model have some significance and not conveying why it is not being optimized or even not been give the value is certainly no one will accept. My suggestion would be, if you not intent to optimize all parameter then you must convey the reason and give them the default set of values used for those parameters. You need to show clearity on the purpose.

I hope this helps.

Cheers.

[Qiang Wang]

HI, Shashank,

Thank you for your kindy explanation. I intent to optimize the hyperparameters using bayesian optimization. I would like to potimize learning rate,epoches,momentum and number of neurons.

is this reasonable? Do you have any better approach?

Thank you!

[Shashank Gupta]

Yes, its resonable to optimize, actually it is very important to optimize all of them. There are many ways to do it, MATLAB has a bayesian approach too. Check this out. It might help you.

[Qiang Wang]

Thank you!