Is calling train with 20 epochs of learning the same as 20 calls to train with 1 epoch of learning?

Hi there.

I've been exploring the train function and I'm a little unsure of how it's operating.

From my understanding, setting net.trainParam.epochs=20 should produce the same network as 20 calls to train with net.trainParam.epochs=1, using the updated network each time. But, when I do this I get very different results. The network produced using net.trainParam.epochs=20 performs much better than the network produced by making 20 calls to train with net.trainParam.epochs=1. Is something special happening in train with multiple learning epochs?

Thanks

-- Jeff

Answer 1

[Greg Heath]

Every time train is called it starts with new values for training parameters like mu, etc. Therefore, if there was someway to specify initial learning parameters, the final values could be extracted and used as initial values for the next call.

Hope this helps.

Thank you for formally accepting my answer.

Greg

[jeffrey cockburn]

Hi Greg. Do you know if it's possible to a) extract the training parameters (mu etc...) and b) specify initial parameters? I can't see a way to get at this.

Thanks

-- Jeff

[Greg Heath]

You can extract them from (no semicolon)

net = net

and/or

tr = tr

However, you would have to modify train in order to use them for initializing the next call.

Maybe it will be modified in newer versions.

Thanks for formally accepting my answer.

Greg

[Ekaterina Kryuchkova]

Hi Greg,

I know this has been asked in 2012, and I was wondering if this is available in the version that is out now. Thank you so much!

Answer 2

[Sean de Wolski]

I'm pretty sure that each call to train nullifies all previous training.

Not quite positive though, I'll wait for Greg to chime in.

[jeffrey cockburn]

Hi Sean.

Well, it look like the network still learns when calling train repeatedly, it just does so more slowly. It doesn't look like train is starting with a fresh network each time.

But, it does look like there's some advantage to giving more epochs of learning for each training call. Are there inter-epoch learning benefits within a call to train (learning rate adjustments, gradient info etc...)?

Thanks again.

-- Jeff

Answer 3

[Greg Heath]

I've been successful achieving the same weights with continuous training vs interupted training with net.trainParam.epochs = 1 when no validation subset is used during training.

The best results are obtained when the number of hidden nodes is not larger than necessary to obtain the a training goal of MSE = 0.01*mean(var(target')).

Hope this helps.

Greg