How to calculate confidence intervals with neural network prediction?

Hello, I would like to present my neural network result with confidence intervals. Because I think interval estimation is better than point estimation. Correct me if i am wrong. I heard that it can be finished with "Neural Network Time Series Prediction". And can I make it with function fitting(fitnet)? or present with prediction interval instead of?

My neural network function is following:

    % Solve an Input-Output Fitting problem with a Neural Network
    % Script generated by Neural Fitting app
    % Created 29-Oct-2015 00:45:44
    %
    % This script assumes these variables are defined:
    %
    %   input - input data.
    %   target - target data.

    x = input';
    t = target';

    % Choose a Training Function
    % For a list of all training functions type: help nntrain
    % 'trainlm' is usually fastest.
    % 'trainbr' takes longer but may be better for challenging problems.
    % 'trainscg' uses less memory. Suitable in low memory situations.
    trainFcn = 'trainlm';  % Levenberg-Marquardt backpropagation.

    % Create a Fitting Network
    hiddenLayerSize = 9;
    net = fitnet(hiddenLayerSize,trainFcn);

    % Choose Input and Output Pre/Post-Processing Functions
    % For a list of all processing functions type: help nnprocess
    net.input.processFcns = {'removeconstantrows','mapminmax'};
    net.output.processFcns = {'removeconstantrows','mapminmax'};

    % Setup Division of Data for Training, Validation, Testing
    % For a list of all data division functions type: help nndivide
    net.divideFcn = 'dividerand';  % Divide data randomly
    net.divideMode = 'sample';  % Divide up every sample
    net.divideParam.trainRatio = 70/100;
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 15/100;

    % Choose a Performance Function
    % For a list of all performance functions type: help nnperformance
    net.performFcn = 'mse'; % Mean Absolute Percentage Error

    % Choose Plot Functions
    % For a list of all plot functions type: help nnplot
    net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
        'plotregression', 'plotfit'};

    % Train the Network
    [net,tr] = train(net,x,t);

    % Test the Network
    y = net(x);
    e = gsubtract(t,y);
    performance = perform(net,t,y)

    % Recalculate Training, Validation and Test Performance
    trainTargets = t .* tr.trainMask{1};
    valTargets = t .* tr.valMask{1};
    testTargets = t .* tr.testMask{1};
    trainPerformance = perform(net,trainTargets,y)
    valPerformance = perform(net,valTargets,y)
    testPerformance = perform(net,testTargets,y)

    % View the Network
    view(net)

    % Calculate Mean Absolute Percentage Error
    err = t-y;
    errpct = abs(err)./t*100;
    MAE = mean(abs(err));
    MAPE = mean(errpct(~isinf(errpct)));

    figure 
    plot(err); 
    title(sprintf('MAPE: %0.3f%%',MAPE))
    xlabel('Days'); ylabel('Error between Actual and Forecasting');

    figure
    plot(y,'DisplayName','Forecasting','LineStyle','--');hold on;plot(target,'DisplayName','Actual');hold off;
    title('Comparing of Actual Customers and Forecasting Customers')
    xlabel('Days'); ylabel('Number of Customers');

    % Plots
    % Uncomment these lines to enable various plots.
    %figure, plotperform(tr)
    %figure, plottrainstate(tr)
    %figure, ploterrhist(e)
    %figure, plotregression(t,y)
    %figure, plotfit(net,x,t)

    % Deployment
    % Change the (false) values to (true) to enable the following code blocks.
    % See the help for each generation function for more information.
    if (true)
        % Generate MATLAB function for neural network for application
        % deployment in MATLAB scripts or with MATLAB Compiler and Builder
        % tools, or simply to examine the calculations your trained neural
        % network performs.
        genFunction(net,'myNeuralNetworkFunction');
        y = myNeuralNetworkFunction(x);
    end
    if (false)
        % Generate a matrix-only MATLAB function for neural network code
        % generation with MATLAB Coder tools.
        genFunction(net,'myNeuralNetworkFunction','MatrixOnly','yes');
        y = myNeuralNetworkFunction(x);
    end
    if (false)
        % Generate a Simulink diagram for simulation or deployment with.
        % Simulink Coder tools.
        gensim(net);
    end

Thanks alot.

Answer 1

[Greg Heath]

When my work has required prediction intervals I have just used

      [ y - sqrt(mse) , y + sqrt(mse) ]

where

 y = net(x)

Hope this helps.

Thank you for formally accepting my answer

Greg

PS Your following label is incorrect:

 net.performFcn = 'mse'; % Mean Absolute Percentage Error

[J1]

Very appreciate to Greg. It is helpful. I see.I will edit it, Thanks a lot.

[Hugo Mendonça]

Greg,

I have a doubt about your answer. You said that one can use the RMSE value as a confidence band, but how could you define it in usual ways as 95% or 90% ?

Regards.

[Greg Heath]

 1. Standardize the errors to zero-mean/unit-variance
 2. Sort the errors to form a cumulative probability
 distribution
 3. Assume the errors are Gaussian distributed and use
 a table.

[Mohsen]

Greg,

thanks for your explanation. But shouldn't we consider then 2 times RMSE (which is actually the standard deviation of the error) for 95% confidence and 3 times RMSE for 99.7% confidence? Considering an interval of plus-minus RMSE give a confidence of only about 68.3%.

Thanks in advnace.

[Greg Heath]

The problem with trying to get too precise is that too many assumptions are made. That is why instead of picking a magic theoretical coefficient times the SCALE DEPENDENT rmse, I prefer to use the normalized mse associated with the coefficient of determination, Rsq (See Wikipedia)which is interpreted as the fraction of target variance that is "explained" by the net

 NMSE = mse(error)/mean(var(target',1))

 Rsq = 1 - NMSE

Five such values can be obtained:

 1. Training
 2. DOFA Training (Degree-of-freedom-adjusted)
 3. Validation
 4. Testing
 5. Overall

Moreover, each can be tabulated and/or plotted w.r.t. multiple designs obtained from varying the number-of-hidden-nodes and number of random-weight-initializations.

Hope this helps.

Greg

[Alex]

Thank you Greg for the answer. But, I got some queries on that. While, y is an array and mse is a number, how is it possible to calculate the PI? Or, maybe I'm missing something.

P.S.: Sorry for my ignorance.

[Salma Hassan]

what about this code:

x= predicted score;

CI= mean (x) +- 1.96*standard error

another question does the confident interval should be between 0 and 1. and if i got (0.45,0.67) what does it mean?