How to implement a custom performance function in R2013b

3 views (last 30 days)
Good input from Amaria in 2013 (<>)
I've replicated the fundamental steps before making ANY changes to the copy of mse (mymse.m and associated copy of +mse package of functions). I found that the steps with the unmodified mymse.m function do not execute correctly.
Here's the steps I followed (per Amaria's post):
copyfile([matlabroot '\toolbox\nnet\nnet\nnperformance\mse.m'], 'mymse.m');
copyfile([matlabroot '\toolbox\nnet\nnet\nnperformance\+mse'], '+mymse');
[x,t] = simplefit_dataset;
net = fitnet(10,'trainscg');
net.performFcn = 'mymse';
net = train(net,x,t);
The training executes but the results show there is no fitting. The resulting model is a flatline. Its a simple test to try. Anybody have success?
As a related test, I used the mae as follows:
net = fitnet(10,'trainscg');
net.performFcn = 'mae';
net = train(net,x,t);
And this trained successfully. So it appears the performance functions provided by the NN Toolbox work, but user-defined performance functions dont work using the procedure described above by Amaria.
I could be doing something wrong, but I've carefully tried several times and cannot get mymse to work. Again, mymse is unchanged code from mse.
Any help is very much appreciated!

Answers (2)

Craig on 15 May 2017
The first two copyfile commands are great, they pull the necessary functions into your current working directories. From there if we execute and follow the debug information we can discover the following.
The 'nn7' argument can be fed to train() or to net() but it appears it is ONLY essential for evaluation using net(). This is because when we set net.performFcn = 'mymse' the nntoolbox will automatically pick up our performance function and will use it for training (you can see it listed in the nntraintool). HOWEVER, if you then try to call net() without using the 'nn7' argument it will output all zeros. For whatever reason training detects the correct net.performFcn and uses it, but the evaluation function does not. You can see this from the command line or from the nntraintool which both show incorrect outputs of zero if net is just called blindly without the 'nn7' argument (as it is if the push buttons in the nntraintool GUI are used without alteration).
Another interesting aspect is that examining the templates for the cost functions it appears as if nn7 (or 'nnet7') is only currently implemented with mse, mae, msesparse, sae and sse cost functions, assuming that is what the 'nnet7' is doing. I don't know this for sure at this time.
If we stop the code when running inside the 'mymse' function, we see that initially the INFO data is setup and made persistent, and this carries details about which functions to use in evaluating various calls to performance. Presumably, if we want to modify the cost function (or override it) then this involves adjusting all these functions suitably. So we need to examine how we need to modify the following subfunctions,
@mymse.apply @mymse.backprop @mymse.forwardprop @mymse.dperf_dwb
Each of these functions can be found beneath the '+mymse' directory we just copied to our working directory. There appear to be other related functions there too (such as perfwb.m and perfw_to_ew.m and it may be necessary to modify them as well for at least consistency sake).

Chibuzo Nnonyelu
Chibuzo Nnonyelu on 22 Apr 2015
Hi Calvin, I am interested in using a custom performance function in MATLAB NNtool and I have been searching to no success. Might you have gotten a solution to your problem? Please help me out, if you have. Thank you.
Gokhan Demirkiran
Gokhan Demirkiran on 10 Feb 2017
You shold use: net = train(net,x,t,nn7); nn7 is the key here! for more information:

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by