SSD Object Detector training results in NaN loss and RMSE

Hello

I've create an SSD with mobilenetv2 with the example from "Create SSD Object Detection Network". But changed the class count to just 1.

For training I've used the sample from "Object Detection Using SSD Deep Learning".

|=======================================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Mini-batch  |  Mini-batch  |  Base Learning  |
|         |             |   (hh:mm:ss)   |     Loss     |   Accuracy   |     RMSE     |      Rate       |
|=======================================================================================================|
|       1 |           1 |       00:00:01 |       3.1220 |       50.98% |         3.39 |      1.0000e-05 |
|       4 |          50 |       00:00:27 |          NaN |        0.00% |          NaN |      1.0000e-05 |
|       8 |         100 |       00:00:53 |          NaN |        0.00% |          NaN |      1.0000e-05 |
|      11 |         150 |       00:01:19 |          NaN |        0.00% |          NaN |      1.0000e-05 |

Is there something I'm missing? Is the SSD model created in the first sample not an actual working model?

Best regards

Link Sample 1: https://ch.mathworks.com/help/vision/examples/create-ssd-object-detection-network.html

Link Sample 2: https://ch.mathworks.com/help/deeplearning/ug/object-detection-using-ssd-deep-learning.html

Edit: I've tried decreasing the learning rate with no success.

[Sai Bhargav Avula]

can you provide more details like what other changes you made to the code ?

can you share the code that you are working on?

[Daniel Jampen]

The only thing I've changed is the line in "Create SSD Object detection network"

numClasses = 5;

to

numClasses = 1;

I've done this because the "Object Detection using SSD Deep Learning" uses just one class for training. (Vehicle)

When using more anchor boxes with bigger sizes it trains longer without resulting in NaN. But at somepoint NaN will still show up.

To my knowledge tho, anchorbox size and count should only impact accuracy and not resulting in NaN values?

Answer 1

[Ryan Comeau]

Hello,

I do not know the exact thing which may be causing this, but if I had to bed on it, I would check all of the bounding boxes in your data set and make sure they are correctly labelling the objects. If you removed 1 class but left the bounding boxes there it could be finding NaN value in this way. Superimpose the bounding boxed on the image and ensure you have the correct labels and locations of these objects.

Second, the lowest recommended learning rate i've seen in literature(i don't have a specific paper to link her unfortunately) is about 1e-6. Low learning rates like this can cause your network to not converge at all since the weights will never be updated enough. What I recommend is use the learn rate drop schedule that is provided here. Here is a sample of what i've used to achive some satisfactory results.

options = trainingOptions('sgdm',...
    'InitialLearnRate',18.0e-4,...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropFactor',0.7, ...
    'LearnRateDropPeriod',1, ...
    'Verbose',true,...
    'MiniBatchSize',24,...
    'MaxEpochs',8,...
    'Shuffle','every-epoch',...
    'VerboseFrequency',1);

Third, to further gain performance, tune your strides and sizes of convolution kernels, you'll need to adjust this to your specific task i cannot help here.

Hope this helps,

RC

[Daniel Jampen]

Thank you for your insights.

Since the Sample in "Link Sample 2: https://ch.mathworks.com/help/deeplearning/ug/object-detection-using-ssd-deep-learning.html" does work when training with

lgraph = ssdLayers(inputSize, numClasses, 'resnet50');

I assume the training data set is correct.

I will make some test with your other points.

Thank you very much.

[Benjamin]

Hello ,do you solve the problem? I have the same problem now and can't fix it.