Dedicated GPU Memory Usage - Permanently increases every time code is run

Every time I run my code, the dedicated GPU Memory Usage permanently increases. I am using a pretrained neural network called detectTextCRAFT which is built into MATLAB. This error only recently started happening and I have been developing this code for multiple weeks so it is a new problem. Initially I would get this error:

Error using dlnetwork/predict

Layer 'codegen_1': Invalid input data. Layer 'Resize_54_new': Invalid input data. Error using 'predict' in layer nnet.cnn.layer.Resize2DLayer. The function threw an error and could not be executed.

Error in detectTextCRAFT (line 141)

networkOutput = predict(craftNet,Ipreprocessed,'Acceleration',params.Acceleration);

Error in Test7 (line 4)

bbox = detectTextCRAFT(I,CharacterThreshold=0.4);

Caused by:

Error using gpuArray/interp1

Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'.

Error in deep.internal.recording.operations.Interp1Op/forward (line 33)

vq = interp1(x,v,xq,op.Method,extrap);

Error in deep.internal.recording.RecordingArray/interp1 (line 7)

vq = recordNary(x,v,xq,extrap,1,op);

Error in dlarray/interp1 (line 76)

dataVq = interp1(dataX, dataV, dataXq, method, dataExtrap);

Error in images.dlresize.internal.interpSpatialDims>interpAlongSpatialDim (line 52)

out = interp1(1:inputDimSize,in,queryPoints',method);

Error in images.dlresize.internal.interpSpatialDims (line 12)

out = interpAlongSpatialDim(out,spatialDim,start,stride,stop,...

Error in nnet.cnn.layer.Resize2DLayer/predict (line 144)

Z = images.dlresize.internal.interpSpatialDims(X,start,stop,stride,layer.DlresizeMethodName,layer.NearestRoundingMode,scale,inputSpatialDimSize);

Then I used gpuDevice(1) to reset the GPU and then I got this error,

Error using nnet.internal.cnn.dlnetwork/predict

Layer 'Conv_0, Relu_1': Invalid input data. The data no longer exists on the device.

Error in nnet.internal.cnn.dlnetwork/DefaultOptimizationStrategy/propagate (line 78)

[varargout{1:nargout}] = inferenceMethod(net, X, layerIndices, layerOutputIndices);

Error in nnet.internal.cnn.dlnetwork/DefaultOptimizationStrategy/predict (line 28)

[varargout{1:nargout}] = propagate(strategy, net, X, ...

Error in dlnetwork/predict (line 661)

[varargout{1:nargout}] = strategy.predict(net.PrivateNetwork, x, layerIndices, layerOutputIndices);

Error in detectTextCRAFT (line 141)

networkOutput = predict(craftNet,Ipreprocessed,'Acceleration',params.Acceleration);

Error in Test7 (line 4)

bbox = detectTextCRAFT(I,CharacterThreshold=0.4);

Then I redownloaded the detectTextCRAFT add-on and now the current problem has been that the dedicated GPU memory usage permanently increases every time code is run. On top of that, the OCR function that detects the words has been outputting much worse that it is supposed to with lots of empty outputs and inaccuracies everywhere. It used to work much better.

Thank you!

[Anay Naik]

This is the main code:

I = imread(image);

bbox = detectTextCRAFT(I,CharacterThreshold=0.4);

%Character Threshold of 0.4 is default

if bbox>0

Icorrect = cell(1,size(bbox,1));

finalOutput = cell(1,size(bbox,1));

recognizedWords = cell(1,size(bbox,1));

for i=1:size(bbox,1)

roi = bbox(i,:);

Icrop = I(roi(2):roi(2)+roi(4),roi(1):roi(1)+roi(3),:);

Ipreprocess = rgb2gray(Icrop);

Ipreprocess = imadjust(Ipreprocess);

Isegment = imbinarize(Ipreprocess);

Isegment = padarray(Isegment,[15 15],0,'both');

se = strel('square',2);

Icorrect{i} = imerode(Isegment,se);

finalOutput{i} = ocr(Icorrect{i});

recognizedWords{i} = [deblank(finalOutput{i}.Text)];

end

%use this block of codes for corrections and adjustments if needed

%variable recognizedWords created with all words

words = string(recognizedWords)

[Anay Naik]

Now that I have restarted everything and opened MATLAB again, the very first error message about the gpu being out of memory : Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'. Please let me know what this means and how I can solve it. Thanks!

Answer 1

[Anay Naik]

I believe I solved this question, the code is pasted below:

[height, width] = size(Im);

imageSize = 2400;

width = width/3;

x = abs(imageSize - height);

y = abs(imageSize - width);

if height>0 && width>0

if x>y

I = imresize(Im,(imageSize/width));

elseif y>x

I = imresize(Im,(imageSize/height));

else

I = imresize(Im,(imageSize/height));

end

end

The issue was that the inputted images were either too large or that I resized them too large myself. This code shrinks or enlarges an image to the specified 'imageSize' without distorting it. It essentially multiplies both dimensions by the same number which results in one of the dimensions being 2400.

Answer 2

[Joss Knight]

This error means you ran out of GPU memory. I can't reproduce any sort of memory leak in R2022a. It's possible that you are permanently storing some of your results in a growing array? If you are running R2022a you can use whos to see how much GPU memory your workspace variables are using, to make sure they are not growing in size each time you run your code. Alternatively, call clear before running your code again, to see if the memory leak goes away.

[Anay Naik]

I think I have solved my problem. My goal is to extract words out of an image as accurately as possible. I thought that if I increased the size of an image then the program could read it better. I believe that I raised the imresize() value too large and so that is why the GPU error occured. I have now only increased the size of the image 2 times instead of 3 and I have included an if statement in case an image inputted is too large to multiply it by 2. My error is now gone since I made this update but returns if I increase the imresize back to 3 times. Is there a maximum height and width for an image where MATLAB cannot handle it and outputs a GPU error? If so, I need that information to make my code stronger. Thank you for your help!