Neural network: generated function usage

Hello! I'm trying to play with neural networks toolbox. I've tought a network to predict values of

sin(x/1150)+1

function using 10 previous values. I used

x = 1:1:240*60

to generate targets set. Training goes just fine and ntstool asks me whether I want to test my network. I test it using values generated for

x = 1:1:480*60.

It says that MSE is 4e-12. I think it's great. I generate a matrix-only function in order to use this network. Unfortunately, when I try to test it with my own code, results are awful. The net cannot even predict the first 50 values! I think the problem is in my test code, here it is:

x = (240 * 60 - 10):1:(240*60);  % keys
func = @(x) sin(x/100) + 1;  % predicted function
y = func(x);  % expected values

vi = [];  % initial values
start = 1;
for i = drange(start:start+9)
    vi = [vi [y(i)]];
end

x2 = (240 * 60+1):1:(240*60+4000);
y2 = func(x2);  % we will predict these values

y1 = [];
for i = drange(x2)
    [v, vi2] = myNeuralNetworkFunction(i, vi);  % predit a single value
    y1 = [y1 [v]];  % save it
    vi = vi([ 1+1:end 1:1 ]);  % shift left by 1 value
    vi(10) = v;  % and append predicted value to the list
end

plot(x2, y2, '-r');
hold on
plot(x2, y1, '-g');

Could anyone help me?

Answer 1

[Greg Heath]

How many points are needed to uniquely specify a sinusoidal function? Since the general form for a pure sinusoid, A*cos(w*t+phi), contains 3 parameters, I think that you need at least 3 points per period.

Your period is

X = 2*pi*1150; % 7225.7

The number of points you have is

N = 240*60 % 14400

The resulting number of points per period is

PPP = Np/X % 1.9929 < 2

So, I do not think that you have enough data per period.

You say that you can model the training data without error.

Are you sure the model was not overfit and overtrained so that the training data was memorized?

How much training data? Ntrn =

How many outputs? O = 1

How many training equations? Ntrneq = Ntrn*O = Ntrn

How many inputs? I = 1

How many hidden nodes? H= ?

How many unknown weights? Nw = (I+1)*H+(H+1)*O = 3*H+1

Do you have more equations than weights? H < (Ntrn-1)/3 = ?

Hope this helps.

Greg

[Ivan Xanx]

Hello Greg, thanks for your response!

About periods: I train the net with two periods which means 14400 points overall. Or you mean that I need to have as much periods as parameter the function has?

About modelling data: Matlab asked whether I would like to test my network and I generated data set for 1:1:360*60. I tested the net with this data and the result was fine. I guess this might be the problem: the net starts at 0, but at given source it starts at 240*60 - 10. As it successfully predicted values beyond the training set, I think it hasn't memorized it.

What 'training data' means? I train the net with 14400 points and I assumed this to be the training data, but now I'm not sure about that.

The net has just one output neuron.

It has one input and 10 delays.

The amount of hidden nodes differs. I vary it from 10 to 20, everything with the same result.

So, Ntrn = 14 400 O = 1 Ntrneq = 14 400 I = 1 H = 20 Nw = 61 So, I have more equations the weights (61 < 4799).

And what should be my next move? Increase training set? Or hidden layer size?

Here is my network diagram:

And here is test response:

Answer 2

[Greg Heath]

Description uses sin(x/1150)

Function uses sin(x/100)

[Ivan Xanx]

Sorry, my fault. I fixed it and rewrote my code:

x2 = 240*60-9:1:240*60;
y2 = func(x2);

vi = y2;

x3 = 240*60+1:1:360*60;

expected = func(x3);

predicted = [];

for i = drange(x3)
    [v, vi2] = myNeuralNetworkFunction(i, vi);
    predicted = [predicted [v]];
    vi = vi([ 1+1:end 1:1 ]);
    vi(10) = v;
end

plot(x3, expected, '-r');
hold on
plot(x3, predicted, '-g');

Where func is defined as

    @(x)sin(x/1150)+1

This function is available in the workspace and was used both for network training and within my test code. This code produces the following plot:

Although when I test the network with Time Series Tool the network was able to predict the whole func(240*60+1:1:360*60).

If I replace

for i = drange(x3)
    [v, vi2] = myNeuralNetworkFunction(i, vi);
    predicted = [predicted [v]];
    vi = vi([ 1+1:end 1:1 ]);
    vi(10) = v;
end

with just

    predicted = myNeuralNetworkFunction(x3, vi);

the result is even worse:

It seems to me that the problem is somewhere here:

    myNeuralNetworkFunction(i, vi)

especially in the first argument, because I give here an X coordinate, but it seems to be incorrect, as the network was trained to predict Y without any relation to X. It seems that the network just ignores this argument, because no matter what I pass there (0 or func(i)) it produces plot equal to the second one.

Or maybe I've chosen the wrong tool and I shall use newff() with 10 inputs for this task?