Improving performance on my neural network in Matlab

I am a novice at using neural networks, and am using them as part of a project to model the severity of traffic accidents.

I have a database in Microsoft Excel, where each row represents a different accident, and each column is a different attribute of the accident.

The attributes are the independent variables. Some of the attributes are categorical (such as weather), and some are numerical (such as Average Speed at the accident site). As per this link, I have transformed my database so that each possible value for the attribute is a separate column (ie, the accident can occur during the day or night, so I have one column in the database labled "day", which can be 1 if true, or 0 if false. One additional field is "road status", whose value could be either "Dry", "wet", "oily", "muddy", "sand/gravel", "other", or "unknown", so I created five columns which represent the first five roadway conditions, whose values can each be either 0 or 1, and if all five column's values are equal to 0, then the status is "unknown", as to maintain one degree of freedom).

I have a total of 72 columns which represent independent variables, and 6645 observations.

The dependent variables are the accident severity, either Fatal, Severe, or Light. As per the Wine Classification - MATLAB & Simulink Example (link), where there was one column for each of three different wineries, I set up three database contains three columns: "Accident Severity - Fatal", "Accident Severity - Severe" and "Accident Severity - Light". Each field can be 1 or zero, and the sum of the three fields is equal to one.

In the Matlab command window, I've specified the inputs (the 72 columns - 6645 observations, representing 6645 samples of 72 elements), and the targets (the three columns, 6645 observations, representing static data: 6645 samples of 3 elements). Next I've run the Neural Network Toolbox ("nnstart") and selected "pattern recognition", then selected my data and transposed it using "Samples are matrix rows" option, and created the neural network.

My problem is that no matter what changes I make to the number of neurons, or the division of data used for training, validation and testing, I cannot get the MSE or % Error to go down, despite numerous training attempts. I have attached a screenshot of my results.

Can anyone help me by telling me exactly what I must do in order to improve my results? I am looking to achieve a result of 90% accuracy or better.

Thank you!

Answer 1

[Greg Heath]

I do not use shuffling. So, use this code to get a general idea.

% The code below is for net.divideFcn = 'dividetrain'. You will have to modify it if you use validation and/or test subsets.

 [I N ] = size(x)
 [O N ] = size(t)
 Neq  = N*O
 Nw   = (I+1)*H+(H+1)*O
 Ndof = Neq-Nw
%Train the net with random initial weights
 for i=1:I
 xsh      = x;
 ind      = randperm(1:N);
 xsh(i,:) = x(i,ind);
% Retrain duplicate model here for option 4.
 ysh      = net(xsh);
 SSEsh    = sse(t-ysh)
 MSEash   = SSEsh/Ndof
 NMSEash(i) = MSEash/mean(var(t'));
end
 [ maxNMSEa imax ] = max(NMSEash);
% Continue training the net with x(imax,:)effectively deleted.
% Repeat I-1 timesNote:

You will have to play around with finding a good way to "effectively delete" x(imax,:) because you also have to deal with the input weights coming from the imax input node to the hidden layer.

Using [], NaN ,zeros, or mean come to mind. Do not use the shuffled version.

I would first try repmat(mean(x(imax,:),2),1,N) or, since inputs are standardized, zeros(1,N) for both x(imax,:) and IW(:,imax)

Hope this helps.

Greg

Answer 2

[Greg Heath]

Transpose your input and target matrices

 [ I N ] = size(input)
 [ O N ] = size(target)

Thank you for formally accepting my answer

Greg

[David Suchinsky]

When running the neural network pattern recognition tool, I had selected: "Samples are Matrix rows", Greg is that what you were referring to? I don't have any trouble running the program and outputting the network, the problem is that I can't get the results up to 90% accuracy.

What is [ I N ] ? Is that something I need to type into the command window?

[Greg Heath]

I was not aware of the samples are rows option.

[I N ] and [ O N ] are the sizes when samples are columns.

I always check those at the beginning of my command line codes.

Answer 3

[Greg Heath]

With the info you have given, my only advice is to try to reduce the number of inputs.

[David Suchinsky]

Thanks for your answer, I'll try reducing the number of variables.

Do you think that choosing a different training algorithm from Scaled Conjugate Gradient to something else will help? I'm not sure how to do this in the GUI for neural network pattern recognition, or can it only be done with a Matlab script? If by script, can you explain step-by-step?

A second question is: After the network has been output to the Matlab workspace, is there a way to check which fields influence the result, and to what extent? Maybe by checking the link weights for each node? Is there a way to list all the values of the line weights? I want to know which attributes of the accident are most influential on the accident severity (in other words, which independent variables are most influential on the dependent variables). The only other option I can think of is to create a matrix of all the possible combinations of 72 variables and see what happens, but that seems very rudimentary.

Answer 4

[Greg Heath]

The only way to guarantee an optimal reduction is to test all possible input variable combinations. However, this is computationally prohibitive. Therefore suboptimal techniques that are not as computationally prohibitive are used.

The resulting weights of any minimum MSE model are easier to understand if input AND output variables are all standardized to zero-mean/unity-variance

zx = zscore(x',1)'; zt = zscore(t',1)';

You can get a preliminary ranking of input variables using backwards STEPWISE or STEPWISEFIT to obtain a linear model.

A stepwise procedure for any model results if

 1. Train multiple nets with all of the input variables
 2. Keep the net with the minimum degree-of-freedom adjusted MSE, MSEa.
 3. One-by-one, each variable is replaced by either
   a. their mean values (zeros if standardized).
   b. a random reshuffling (RANDPERM) of it's values 
 4. The resulting MSEa is tabulated. 
 5. The input corresponding to the largest MSEa is deleted
 6. The net is retrained without re-initialization 
 7. The process is repeated until either
    a. The desired no. of inputs have been deleted
    b. The maximum allowable MSEa is reached

Alternate approaches:

3b. The random shuffling for each variable is performed multiple times. The mean or median MSEa is used for the ranking.

4. The net is retrained before the resulting MSEa is tabulated.

Hope this helps.

Thank you for formally accepting my answer

Greg

[David Suchinsky]

Hi Greg,

Thank you for your answer. Could you please explain, step by step, how to do 3b. and 4.? What is the Z score used for (or is that in response to my second question about measuring the influence of the independent variables?)?

Additionally, as I had asked previously, is there a way to rank the independent variables depending on how influential they are on the accident severity? I though that maybe looking at the value of the weights on the links would be helpful - do you know if that would work, and how can I do it in Matlab?

Thank you again.