Trained model is not provding the desired output result.

Hi,

I am traing a model for image segmentation. During the trainig both the validation and training accuracy reaches around 92%. But when I evalaute my trained model using test images or even using the trian images the model gives the accuracy around 50%.

What could be the reason of this? Is this probelm is due to overfitting?

[KSSV]

If the accuracy is 92%, for the train images the accuracy should not reduce. It should be on par as it was while training. There might be a mistake while you are predicting from the trained model. We can comment more, on looking at your code.

[Walter Roberson]

There is a famous example of a US Air Force AI project to distinguish pictures of US aircraft from pictures of Russian aircraft. In testing and validation the program achieved 100% success rate -- but when deployed to testers, the program often failed.

It turned out that what the program was really detecting was whether the aircraft was pointed left or pointed right, because the US aircraft used for the training image were all pointing right and the Russian aircraft trained on were all pointing left.

[Shoaib Ali]

I am using the following code to read the data, training the model, and testing the trained model.

%%%%%%%%% loading the trainig, val and testing data %%%%%
addpath 'D:\data\train'
load gTruth % load gTruth file of training data
[imdsTrain,pxdsTrain] = pixelLabelTrainingData(gTruth);
pximdsTrain = pixelLabelImageDatastore(imdsTrain,pxdsTrain);

addpath 'D:\data\val'
load gTruth % load gTruth file of Validiation data
[imdsVal,pxdsVal] = pixelLabelTrainingData(gTruth);
pximdsVal = pixelLabelImageDatastore(imdsVal,pxdsVal); 

addpath 'D:\data\test'
load gTruth % load gTruth file of test data
[imdsTest,pxdsTest] = pixelLabelTrainingData(gTruth);
pximdsTest = pixelLabelImageDatastore(imdsTest,pxdsTest); 

%%%%%%%%%%%%%%%%%% Training %%%%%%%%%%%%
rng(2) % For reproducibility CE
Trained_net= trainNetwork(pximdsTrain,lgraph,options);

%%%%%%%%%%%%%%%%%% Testing the model %%%%%%%%%%%%
pxdsResults = semanticseg(imdsTest,Trained_net, ...
    'MiniBatchSize',16, ...
    'WriteLocation','D:\Test Images Output', ...
    'Verbose',false);

Answer 1

[Kapil Gupta]

I understand you are trying to train a model for image segmentation but you are getting very low testing accuracy as compared to training/validation accuracy.

This generally happens when your model is learning the data instead of learning the pattern, better known as 'Overfitting'.

You can try the following few things:

• Use of regularization technique
• Make sure each set (train, validation and test) has sufficient samples like 60%, 20%, 20% or 70%, 15%, 15% split for training, validation and test sets respectively.
• Perform k-fold cross validation
• Randomly shuffle the data before doing the spit, this will make sure that data distribution is nearly the same.If your data is in datastore you can use 'shuffle' function else you can use "randperm" function.

You can also go through the following MATLAB Answer which discusses a similar issue:

https://in.mathworks.com/matlabcentral/answers/556924-how-to-increase-testing-accuracy-in-cnn