Recognizing text from image of a calculator

Hi,

I have trying to extract the numbers from a calculator screen image. I am using my laptop webcam to detect the image and simultaneously display the result after image processing. Can anybody please help how to proceed??

[sandeep bayi]

iam too working on the like wise project and need help for extracting the numbers from led display of seven segment data. Can anyone please help me?

Answer 1

[Image Analyst]

Very very easy. You don't even need to do sophisticated OCR. You merely need to sample the image at known locations for known numbers. Trivial for a 7 segment display. Just read the image at 7 locations. Let's say the locations are

    1
  2   3
    4
  5   6
    7

Now a "1" would have 2 and 5 lit, and all the others unlit.

A "2" would have 1, 3, 4, 5, & 7 lit and the others unlit.

You can make up a 128 element look up table and use that to read off the number, given an input pattern. it's the same concept but a little more complicated if you have something with more resolution than a 7 segment display.

[Soumyadip]

its fine but my calculator has more resolution than a 7 segment display. Can you suggest how to seperate this individual dots and identify the necessary pattern from the look up table?? Actually I am a total rookie in matlab. It would help if you provide some resources regarding this problem.

[Image Analyst]

It's the same concept. You may just need to have 7 "zones" where you use mean2() to get the mean value of the image in that location. It will either be bright or dark depending on if the number is "set" there. Where did you upload your image?

[Soumyadip]

Actually, my laptop has a VGA webcam where the quality of the pic is not good. Also, the display of the calculator is not at all like 7-segment. If you can please provide me your email id, I can mail you the images. Kindly help.

[Image Analyst]

Try tinypic.com or snag.gy/. You may have to process the image a bit so that you can tell when a segment is lit or not. After that, the concept is the same.

[Soumyadip]

http://i50.tinypic.com/dpvms8.jpg

This is the image. How to proceed?

[Image Analyst]

First of all, improve your image by using more light. Next, make sure your calculator is in the same location every time. Then just use the same concepts I spelled out. There are certain locations that will be blue and certain locations that will be gray. These locations will be different for every number. I don't think you should have to check more than about 7-10 regions to uniquely identify every number. Is there something in the algorithm that is still confusing? Or do you understand the concept and algorithm, but you just don't know how to put it into MATLAB code? Here's a hint. To get the red, green, and blue value you can use mean2:

meanRed = mean2(rgbImage(row1:row2, col1:col2, 1));
meanGreen = mean2(rgbImage(row1:row2, col1:col2, 2));
meanBlue = mean2(rgbImage(row1:row2, col1:col2, 3));
tolerance = 10; % or whatever.
zoneIsBlue(zoneNumber) = meanBlue > (meanRed + tolerance) & meanBlue > (meanGreen + tolerance)

Of course the rows and columns will have to refer to different locations for each number because they are not all in the same location. You could pre-identfy those regions with ginput(), imrect(), or rbbox(), which is easiest if your calculator is in the same location, as I recommended. If the calculator moves around and you, unfortunately, have no way at all to prevent that, then you can identify the blue region and try to find out how many numbers are showing by looking at the width of the blue region, then get your zones knowing that. You can look in my File Exchange for several color segmentation methods.

[Soumyadip]

I tried getting the blue part of the RGB image above but the numbers are not clearly displaying there..Do you mean that the numbers will be the blue part where as the background will be the grey part? Also, can I improve the image so that processing becomes better?

[Image Analyst]

More light and a longer exposure time. And save it as PNG, not jpg. And of course a better camera and lens. Here's some code to get you started:

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
imtool close all;  % Close all imtool figures if you have the Image Processing Toolbox.
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 20;
% Read in a standard MATLAB color demo image.
folder = 'C:\Users\Soumyadip\Documents\Temporary';
baseFileName = 'dpvms8.jpg';
% Get the full filename, with path prepended.
fullFileName = fullfile(folder, baseFileName);
if ~exist(fullFileName, 'file')
  % Didn't find it there.  Check the search path for it.
  fullFileName = baseFileName; % No path this time.
  if ~exist(fullFileName, 'file')
    % Still didn't find it.  Alert user.
    errorMessage = sprintf('Error: %s does not exist.', fullFileName);
    uiwait(warndlg(errorMessage));
    return;
  end
end
rgbImage = imread(fullFileName);
% Get the dimensions of the image.  numberOfColorBands should be = 3.
[rows columns numberOfColorBands] = size(rgbImage);
% Display the original color image.
subplot(2, 2, 1);
imshow(rgbImage);
axis on;
title('Original Color Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);

% Crop the image.
rgbImage = imcrop(rgbImage, [275, 320, 250 80]);
% Display the image.
subplot(2, 2, 2);
imshow(rgbImage);
axis on;
title('Cropped Color Image', 'FontSize', fontSize);

% imwrite(rgbImage, 'calculator.png');

% Extract the individual red, green, and blue color channels.
redChannel = rgbImage(:, :, 1);
greenChannel = rgbImage(:, :, 2);
blueChannel = rgbImage(:, :, 3);

tolerance = 7; % or whatever.
bluePixels = blueChannel > (redChannel + tolerance) & blueChannel > (greenChannel + tolerance);
% Display the image.
subplot(2, 2, 3);
imshow(bluePixels);
axis on;
title('Blue Pixels', 'FontSize', fontSize);

% meanRed = mean2(redChannel(row1:row2, col1:col2));
% meanGreen = mean2(greenChannel(row1:row2, col1:col2));
% meanBlue = mean2(blueChannel(row1:row2, col1:col2));

[Soumyadip]

Thanks immensely for the code. However, I think it would not be possible to keep the calculator always at EXACTLY the same position though I will try my level best. Then I count the number of digits based on width of blue pixels. But how to segement them ? Can you please explain what we are doing by the meanRed, etc variables?

[Image Analyst]

Well I just took you up as far as getting an binary image that says whether the pixels are blue or not. You still have to examine that image in each location of each digit to determine what the digit is. That's what that code is for. You need to determine what all the row and columns are for the 7 locations and the N digits.

[Soumyadip]

Hi...I thought of doing it in a different way using pattern recognition. But first I intend to convert the image into a binary one where only the digits will be black and the rest 0. How to do this? Any image enhancement because normal thresholding is not working?? The link to the cropped image - http://i49.tinypic.com/33y25u0.png

[Image Analyst]

Here, I did it using color classification plus some standard image processing techniques to clean the image of spurious blobs so that only the numbers remain. I looked at the 3D color gamut and noticed that in LAB color space, the numbers had negative b values (bluish) while the background had positive b values (greenish/yellowish). Although it does an excellent job on the one image you provided, you'd still be better off getting a better camera to get less noise and using more light, which will also give less noise.

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
imtool close all;  % Close all imtool figures if you have the Image Processing Toolbox.
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 20;

% Read in a standard MATLAB color demo image.
folder = 'C:\Users\soumyadip\Documents\Temporary';
baseFileName = 'calculator.png';
% Get the full filename, with path prepended.
fullFileName = fullfile(folder, baseFileName);
if ~exist(fullFileName, 'file')
  % Didn't find it there.  Check the search path for it.
  fullFileName = baseFileName; % No path this time.
  if ~exist(fullFileName, 'file')
    % Still didn't find it.  Alert user.
    errorMessage = sprintf('Error: %s does not exist.', fullFileName);
    uiwait(warndlg(errorMessage));
    return;
  end
end
rgbImage = imread(fullFileName);
% Get the dimensions of the image.  numberOfColorBands should be = 3.
[rows, columns, numberOfColorBands] = size(rgbImage);
% Display the original color image.
subplot(2, 2, 1);
imshow(rgbImage);
title('Original Color Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);

% Convert to lab color space.
cform = makecform('srgb2lab');
lab = applycform(im2double(rgbImage),cform);
LImage = lab(:,:, 1);
aImage = lab(:,:, 2);
bImage = lab(:,:, 3);
% Display them
subplot(2,2,2);
imshow(LImage, []);
title('L Image', 'FontSize', fontSize);
subplot(2,2,3);
imshow(aImage, []);
title('A Image', 'FontSize', fontSize);
subplot(2,2,4);
imshow(bImage);
title('B Image', 'FontSize', fontSize);

figure; % Open a new figure.
subplot(2, 2, 1);
imshow(bImage, []);
title('B Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);

% Binarize to get negative b (bluish pixels).
originalBinaryImage = bImage < 0;
subplot(2, 2, 2);
imshow(originalBinaryImage, []);
title('Binary Image', 'FontSize', fontSize);

% Clean up the image.
binaryImage = bwareaopen(originalBinaryImage, 20);
% Dilate to merge regions together.
binaryImage = imdilate(binaryImage, true(9));
% Fill holes to make it more solid;
binaryImage = imfill(binaryImage, 'holes');
subplot(2, 2, 3);
imshow(binaryImage, []);
title('Cleaned Binary Image', 'FontSize', fontSize);

% Find the rows
verticalProfile = sum(binaryImage, 2);
subplot(2, 2, 4);
plot(verticalProfile, 'b-');
grid on;
title('Vertical Profile', 'FontSize', fontSize);
% Find the top and bottom rows - where the jumps are biggest.
[~, topRow] = max(diff(verticalProfile))
[~, bottomRow] = max(-diff(verticalProfile))
% Crop image - erase above top row and below bottom row
binaryImage(1:topRow,:) = false;
binaryImage(bottomRow:end,:) = false;

% Find the columns
horizontalProfile = sum(binaryImage, 1);
figure
subplot(2, 2, 1);
plot(horizontalProfile, 'b-');
grid on;
title('Horizontal Profile', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);
% Find the top and bottom rows - where the jumps are biggest.
[~, leftCol] = max(diff(horizontalProfile))
[~, rightCol] = max(-diff(horizontalProfile))
% Crop image - erase above top row and below bottom row
binaryImage(:, 1:leftCol) = false;
binaryImage(:, rightCol:end) = false;
subplot(2, 2, 2);
imshow(binaryImage, []);
title('Cleaned Binary Image', 'FontSize', fontSize);

% Extract the largest blob
% Find the areas.
labeledImage = bwlabel(binaryImage);
blobMeasurements = regionprops(labeledImage, 'area');
allAreas = [blobMeasurements.Area];
% Sort in descending order
[sortedAreas, sortIndexes] = sort(allAreas, 'descend');
% Extract the blob with the largest area.
maskImage = ismember(labeledImage, sortIndexes(1)) > 0;
% Mask this against the first binary image.
originalBinaryImage(~maskImage) = false;
subplot(2, 2, 3);
imshow(originalBinaryImage, []);
title('Cleaned Binary Image', 'FontSize', fontSize);

[Soumyadip]

Thanks sir immensely for the code. The conversion to the Lab color space proved really useful. Although I have a few doubts:

1) While finding the columns where the jumps are biggest, the rightmost col did not give the desired value because there is a sharp decline in the horizontal profile at X = 129, so many of the digits got clipped. How to correct this?

2) If I wish to extract the digits directly from the bImage by cleaning up some noise, is it feasible?

[Soumyadip]

I solved the first problem sir.. I increased the structuring element in the imdilate function from true(9) to true(11). It works now.

[Soumyadip]

Now I have a problem. I have to isolate only the calculator screen (as I attached in the link above) from the main image automatically every time. I tried that using bwboundaries but it is only thresholding those regions. How to extract a particular region??

Main image - <http://i46.tinypic.com/28cd4dl.jpg>

[Image Analyst]

The link does not work. Plus I don't know what you mean about bwboundaries() and thresholding. You may have to tweak the algorithm as you encounter new and different images. In fact there are some images where it will fail completely, like where there are no digits at all or where the noise level is too great, etc.

[Soumyadip]

Sorry the pic is

Please suggest how to extract the screen only.

[Image Analyst]

I'd just threshold to find the LAB value of the screen.

[Soumyadip]

I didnt quite understand. How to find LAB value of the screen?

[Image Analyst]

Recall my code above:

% Convert to lab color space.
cform = makecform('srgb2lab');
lab = applycform(im2double(rgbImage),cform);
LImage = lab(:,:, 1);
aImage = lab(:,:, 2);
bImage = lab(:,:, 3);

Just threshold it properly. Pick a row and columns - say put "axis on" or something and look at the lab images to see about what lab value the screen has and threshold to select that.

[Soumyadip]

I tried thresholding the b image but the screen is not getting isolated at all. Actually the screen is not having any distinct property and many corners are also getting thresholded. Can you please provide me a little more code?

[Image Analyst]

Sorry but I think I've provided you enough to go on. If I do this, then there will be another problem with another weird image, and I can't go on to do the whole project with you. And I don't have time to delve deeply into any one particular project. I think you have the general concepts enough for you to continue on your own and take pride in ownership of your code. If lab thresholding won't work, try thresholding on RGB and look for large black squares. Of course this won't work the minute you start to look at black calculators instead of white calculators - see what I mean?

[Soumyadip]

okay...guess i am on my own..thanks