Unknown Class in an Image Classification Problem

Hello,

I am using a pretrained model to classify my images (SqueezeNet). I have a database of images of n classes each one in a folder labeled by its name.

If I give an unknown object or scene image as an input, it will be classified as one of the n classes, which is clearly misleading!

How to determine "unknown class", knowing that I am using a pretrained model, instead of attributing a wrong class?

Thank you!

Answer 1

[Image Analyst]

Did you use classify()?

[YPred,scores] = classify(net,imds) predicts class labels for the image data in imds using the trained network, net.

You can look at the scores and if the score is too low, just call it unclassified instead of using the suggested classification.

[Andrea Daou]

Yes, I tried to do it using this method after asking the question. Do you think it is better to compare the difference between the classes scores to a defined amplitude or to look into the max(scores). Or, it will lead to the same result?

Thank you!

[Image Analyst]

What I would do it look at the images that you thought should be "unclassified" and see what their scores are. Then compare that histogram to the histogram of scores from images that you thought should be classified. You might find a value for score that differentiates unclassified from classified. Like if the unclassified images all have scores less than 0.7 and classified ones are all more than 0.7, just reclassify any image with a score of less than 0.7 from whatever class it suggested to "unclassified" or "unknown".

[Andrea Daou]

Thank you!

Answer 2

[Ameer Hamza]

This is not a simple problem as it may seem. You can think of its like an attack/fool the model to give a wrong class. Read about recent papers related to adversarial machine learning and fooling attacks on deep learning models. One way to deal with this problem is to develop a network with n+1 classes and assign last class the label of "unknown class". Then you need to train the network with the correct images and the non-expected images. Nevertheless, it is currently an active field of research, and a lot of work is being done related to this. It cannot be trivially solved using some built-in functions. Read this paper for some context about this issue: https://arxiv.org/pdf/1909.08072.pdf. Especially refer to section 4.2.2, which mentions training the network using adversarial examples.

[NASRIN AKTER]

Hello

If I do a binary classification (classes 'a' and 'b') and then test with just one class (say, class 'a'), would that be wrong or confusing for the network?

[Image Analyst]

@NASRIN AKTER no that would not confuse the network. Once you've trained the network, as long as you pass in an image of the proper dimensions it doesn't really care what you put into it. It's not "confused" although it may report your image as the wrong class (class b) for some images that are not clearly of class "a" type. Or if the image looks nothing like either class a or class b (let's say it is really from class c but you didn't give it any images from class c to train on), it will be forced to pick one class, which of course will be the wrong class.

But not sure what you mean by "confuse". Like I said, it will work but it may give an answer that doesn't necessarily agree with what you would say.