try...catch for memory full error in imaq toolbox ?

Hi,

I am recording data with an usb3vision camera. I am directly capturing to RAM. After a while, the memory of the computer is full and capturing stops. I would like to catch this error and display a msgbox. This is the code I am testing:

clc

imaqreset

delete(imaqfind); %clears all previous videoinputs

hwinf = imaqhwinfo;

info = imaqhwinfo(hwinf.InstalledAdaptors{1});

industrial_camera_name = info.DeviceInfo.DeviceName;

industrial_camera_supported_formats = info.DeviceInfo.SupportedFormats;

industrial_camera_vid = videoinput(info.AdaptorName,info.DeviceInfo.DeviceID,'Mono8');

industrial_camera_settings = get(industrial_camera_vid);

industrial_camera_settings.Source.ExposureTime =20;

industrial_camera_vid.FramesPerTrigger = 10000;

flushdata(industrial_camera_vid);

try

start(industrial_camera_vid);

%wait(industrial_camera_vid);

while industrial_camera_vid.FramesAcquired < industrial_camera_vid.FramesPerTrigger

pause(1)

disp(num2str(industrial_camera_vid.FramesAcquired))

end

catch ME

imaqreset

if strcmpi(ME.identifier,'imaq:imaqmex:outofmemory')

msgbox('Out of memory. RAM is full, most likely, you need to lower the amount of frames to capture to fix this error.','modal');

end

if strcmpi(ME.identifier,'imaq:wait:timeout')

msgbox('Image capture timeout. Most likely, memory is full and you need to lower the amount of frames to capture to fix this error. It is also possible that the synchronization cable is not plugged in correctly.','modal');

end

end

The problem is that the while loop never stops, even if the image capture crashes due to full RAM. How can I stop the while loop and display a proper error message?

This is how my output looks like, the while loop keeps on running:

3870

4031

4192

4353

4514

4675

Error event occurred at 16:38:40 for video input object: Mono8-gentl-1.

Unable to allocate memory for an incoming image frame due to insufficient free physical memory.

Unable to allocate memory for an incoming image frame due to insufficient free physical memory.

Error in memory_full (line 18)

pause(1)

4754

4754

4754

4754

4754

Answer 1

[Image Analyst]

Perhaps put the try catch INSIDE the while loop. Then in the catch, put a break to kick it out of the loop if an error was thrown.

start(industrial_camera_vid);
while industrial_camera_vid.FramesAcquired < industrial_camera_vid.FramesPerTrigger
    try
        %wait(industrial_camera_vid);
        pause(1)
        disp(num2str(industrial_camera_vid.FramesAcquired))
    catch ME
        % Some error occurred if you get here.
        imaqreset
        if strcmpi(ME.identifier,'imaq:imaqmex:outofmemory')
            uiwait(msgbox('Out of memory. RAM is full, most likely, you need to lower the amount of frames to capture to fix this error.','modal'));
        end
        if strcmpi(ME.identifier,'imaq:wait:timeout')
            uiwait(msgbox('Image capture timeout. Most likely, memory is full and you need to lower the amount of frames to capture to fix this error. It is also possible that the synchronization cable is not plugged in correctly.','modal')));
        end
        break; % out of while loop.
    end
end

[dpb]

Maybe, but...

My guess is that since the camera is collecting data in the background, this is happening all internal to that code that the start(industrial_camera_vid) began the acquisition and the error isn't actually throwing an error but is just an "error condition" internal to the acquistion.

Methinks will need to see if there's not a way to interact with the hardware and query the state and have it generate another callback or somesuch.

Or, monitor the free memory or compute a priori how much memory is available and knowing the size of the images and associated overhead, limit the number of frame acquisitions to begin with.

Bound to be a way to interact with the device, but w/o the documentation, we don't know the details...

[William Thielicke]

Hi, thanks for the ideas. Putting the try catch inside the loop doesn't seem to make a difference

2262

2424

2585

2746

2907

3068

Error event occurred at 19:57:05 for video input object: Mono8-gentl-1.

Unable to allocate memory for an incoming image frame due to insufficient free physical memory.

Unable to allocate memory for an incoming image frame due to insufficient free physical memory.

Error in memory_full (line 38)

pause(1)

3152

3152

3152

[William Thielicke]

Monitoring the free memory doesn't work somehow. There doesn't seem to be a relation between the numbers that I get from the memory command and the free RAM. At least I can't see any relation. The windows task manager seems much clearer when I want to predict when the RAM is full.

[William Thielicke]

But as you suggested, I might be able to modify the error handling of the image acquisition object:

https://de.mathworks.com/help/imaq/errorfcn.html

I don't understand how, but I'll try to find a way...

[Image Analyst]

Are you saying that it's actually the call to pause(1) that is throwing the eror?

Error in memory_full (line 38)

pause(1)

[William Thielicke]

No, I don't think so. It seems to be a general error, maybe all RAM is used and pause can't be executed anymore?

[dpb]

All the out of memory error messages are coming from the acquisition control code, looking to be a routine called memory_full that is where it (most likely) tries to do a low-level malloc() call and finds a NULL pointer on return. Unless the system has a way you can tell it to do something other than just fail and return with something you can get your hands on with the controlling code (and it would seem there must be some way), then you're stuck.

Trying to monitor free contiguous memory from the MATLAB memory is fraught with issues as Bruno notes below, but barring being able to do something with the hardware interface itself it may be about as good as can do.

I thought perhaps there would be a neat way with PowerShell but it doesn't seem to help all that much to be any cleaner. The command shell replacemant I use (Take Command Console from JPSoftware, shameless plug to a super product) has an executable builtin OS function that works as

C:\...> echo %@winmemory[2]
8932417536

that returns almost identically the same number as does the MATLAB memory command as Bruno wrote it out...

>> [~,sys]=memory;fprintf('%0.f\n',sys.PhysicalMemory.Available)
8951324672
>> 

so is probably just as well to go that way as to spawn the other process.

[dpb]

"I don't understand how, but I'll try to find a way..."

Find the default imagcallback function and look at it -- you'll see it's what is outputting the error messages you're getting; set a breakpoint in it and you'll be able to see the object and data that it receives.

Use the function as a pattern; I think what you'll want to do is to have your replacement return a flag variable that can test inside the while loop and break as @Image Analyst suggests; what the default does is just display the error info to the command window; you need it to do something more than that; return a value that you can check and take action on.

Or, depending upon the needs of the application; you could have it do the stop acquisition and so on there; just that you would then probably(?) need a global parameter to use as the control variable for the while loop to turn it off...just what/how to construct it depends on what is wanted to be done.

[William Thielicke]

Thanks, I'll check out the original error function and adapt it. I'll also set a global flag to break the while loop, this should work.

Answer 2

[Bruno Luong]

To my knowledge there is no simple way to detect memory issue.

What you ask MATLAB, e.g., using command such as

[~,sys] = memory;
membytes = sys.PhysicalMemory.Available; % bytes  

I will always report a positive number, even if there is memory runs out since it will swap on HD if you requires MATLAB to allocate array. And when there is NO physical RAM available for instanant the acquisition SW to run correctly there is buffer overrun and it can result crashes. The best thing is to monitor membytes long before and detect when it is no longer reduced and low. It is a sign that the RAM starts to runout and you have to anticipate and stop the acquisition before it crashes.

Answer 3

[William Thielicke]

I am getting a bit closer, I am able to display a messagebox with a suitable hint:

clc

imaqreset

delete(imaqfind); %clears all previous videoinputs

hwinf = imaqhwinfo;

info = imaqhwinfo(hwinf.InstalledAdaptors{1});

industrial_camera_name = info.DeviceInfo.DeviceName;

industrial_camera_supported_formats = info.DeviceInfo.SupportedFormats;

industrial_camera_vid = videoinput(info.AdaptorName,info.DeviceInfo.DeviceID,'Mono8');

industrial_camera_settings = get(industrial_camera_vid);

industrial_camera_settings.Source.ExposureTime =20;

industrial_camera_vid.FramesPerTrigger = 10000;

flushdata(industrial_camera_vid);

industrial_camera_vid.ErrorFcn = @CustomIMAQErrorFcn;

start(industrial_camera_vid);

while industrial_camera_vid.FramesAcquired < industrial_camera_vid.FramesPerTrigger

pause(0.1)

[~,sys] = memory;

membytes = sys.PhysicalMemory.Available;

disp(['Frames captured: ' num2str(industrial_camera_vid.FramesAcquired) ', Memory: ' num2str(membytes)])

end

function CustomIMAQErrorFcn(~,error_cause)

%imaqreset %when this is executed, Matlab crashes instantly...

disp('Error during capture.')

if strcmpi(error_cause.Data.MessageID,'imaq:imaqmex:outofmemory')

msgbox('Error, RAM is full, reduce nr. of frames to capture.','modal')

else

msgbox(['Error, some other error occured: ' error_cause.Data.MessageID],'modal')

end

end

But I wonder if there is a way to stop the while loop from my custom "CustomIMAQErrorFcn"?

This is the output:

Frames captured: 3437, Memory: 3099373568

Frames captured: 3482, Memory: 2996944896

Frames captured: 3526, Memory: 2891186176

Frames captured: 3570, Memory: 2786598912

Frames captured: 3615, Memory: 2678345728

Frames captured: 3660, Memory: 2572312576

Frames captured: 3706, Memory: 2464108544

Frames captured: 3752, Memory: 2357243904

Frames captured: 3799, Memory: 2248380416

Frames captured: 3845, Memory: 2136969216

Frames captured: 3892, Memory: 2027577344

Frames captured: 3938, Memory: 1915494400

Frames captured: 3985, Memory: 1808896000

Frames captured: 4002, Memory: 1708388352

Error during capture.

Unable to allocate memory for an incoming image frame due to insufficient free physical memory.

Error in memory_full (line 22)

pause(0.1)

Frames captured: 4002, Memory: 1694830592

Frames captured: 4002, Memory: 1694830592

Frames captured: 4002, Memory: 1689645056

Frames captured: 4002, Memory: 1689632768

Frames captured: 4002, Memory: 1689620480

There still doesn't seem to be a way to predict the crash from the remaining memory...

[Bruno Luong]

What surprise me in your log is that the acquisition crashes when free memory is 1.7 Gb. It's a wild guess but it looks to me like the whole video buffer are reallocate each time a new frame arrive. Are you surre the acquisition is called correctly as by the vendor recommendation?

If the buffer is reallocated then you should not monitor the RAM, but the largest contiguous block avalable as dpb suggests.

[William Thielicke]

Thanks I'll check this!

Answer 4

[dpb]

Well, you probably can manage to do it from the above, but it's peculiar as Bruno notes...observe if we take your monitoring data and plot it to see what it looks like ("A picture is worth..." even when collecting images. <vbg>)

s=["Frames captured: 3437, Memory: 3099373568"
"Frames captured: 3482, Memory: 2996944896"
"Frames captured: 3526, Memory: 2891186176"
"Frames captured: 3570, Memory: 2786598912"
"Frames captured: 3615, Memory: 2678345728"
"Frames captured: 3660, Memory: 2572312576"
"Frames captured: 3706, Memory: 2464108544"
"Frames captured: 3752, Memory: 2357243904"
"Frames captured: 3799, Memory: 2248380416"
"Frames captured: 3845, Memory: 2136969216"
"Frames captured: 3892, Memory: 2027577344"
"Frames captured: 3938, Memory: 1915494400"
"Frames captured: 3985, Memory: 1808896000"
"Frames captured: 4002, Memory: 1708388352"];
f=str2double(extractBetween(s,'ed: ',','));
m=str2double(extractAfter(s,'ry: '));
tCapture=table(f,m,'VariableNames',{'Frame','FreeMemory'});
tCapture=addvars(tCapture,[nan;diff(tCapture.Frame)],[nan;-diff(tCapture.FreeMemory)],'NewVariableNames',{'NFrames','MemUsed'});
tCapture.MemPerFrame=tCapture.MemUsed./tCapture.NFrames/1024/1024

tCapture = 14×5 table
    Frame    FreeMemory    NFrames     MemUsed      MemPerFrame
    _____    __________    _______    __________    ___________

    3437     3.0994e+09      NaN             NaN         NaN   
    3482     2.9969e+09       45      1.0243e+08      2.1707   
    3526     2.8912e+09       44      1.0576e+08      2.2923   
    3570     2.7866e+09       44      1.0459e+08      2.2669   
    3615     2.6783e+09       45      1.0825e+08      2.2942   
    3660     2.5723e+09       45      1.0603e+08      2.2471   
    3706     2.4641e+09       46       1.082e+08      2.2433   
    3752     2.3572e+09       46      1.0686e+08      2.2155   
    3799     2.2484e+09       47      1.0886e+08      2.2089   
    3845      2.137e+09       46      1.1141e+08      2.3098   
    3892     2.0276e+09       47      1.0939e+08      2.2197   
    3938     1.9155e+09       46      1.1208e+08      2.3237   
    3985     1.8089e+09       47       1.066e+08       2.163   
    4002     1.7084e+09       17      1.0051e+08      5.6383   

yyaxis left
plot(tCapture.Frame,tCapture.FreeMemory/1024^3,'X')
ylabel('Free Memory (GB)')
yyaxis right
plot(tCapture.Frame,tCapture.MemPerFrame,'X')
hAx=gca;
set(hAx.YAxis,{'TickLabelFormat'},{'%0.1f'})
title('Image Capture Memory Useage vs Frame Number')
ylabel('Memory Per Frame(MB)')
xlabel('Frame Number')

We can observe the memory usage is almost perfectly linear with frame number until the very end(*) when the indicated memory usage almost doubled on average. We don't have the data for every image to observe, but one presumes that probably it stayed at the 2MB value until the very last acquired frame and then ran into the allocation problems.

As Bruno, it does seem peculiar that the system still has almost 2GB of overall free memory, but can't seem to find 2MB free; this may indicate some per process limit or other issue with how memory usage is being handled.

(*) Looking at the finer detail, NB the memory per frame is pretty flat until after frame 3800, it then seems more variable for some reason. It's also tantalizing that the first point shown is also lower before the plateau is established after 3500; it would be interesting to observe what the startup transient might look like and if for some reason the application is requiring more memory as it goes along on a per image basis.

We don't know the size/depth of the frame so don't know how this compares to an expected memory footprint.

[Bruno Luong]

interesting. Unfortunately when the Memory Per Frame goes up, the error is already occurs. So it cannot be used to anticipate the error.

My guess is that the acquistion does this for every frame. It has a video buffer that grow

1. Allocate a new video buffer (n+1) frames
2. Copy the old buffer to the new buffer
3. Get a latest frame in the last image block
4. Free the old buffer

If step 1 fails the process stop and two buffers (one complete one partial) are allocated, therefore wee see double Memory Per Frame after the error.

I don't know but it seems this acquisition SW is not well designed if it works similar to what I have described.

[dpb]

Yeah, that would be a bummer, indeed...but it does seem like that is quite likely the way it is implemented given the observed behavior.

[William Thielicke]

So I guess you don't have access to the IMAQ toolbox? Indeed I can only capture around 4000 images on a 16GB RAM machine. The images are approx. 2.2 Mégapixel at 8bit grayscale. So one image is approx. 2.2 MB (as you calculatd already) resulting in 8.8 GB memory. There are also ways to capture to an AVI file with the IMAQ toolbox. But the rest of the program needs to use uncompressed single images, and the conversion from Avi to images takes a veeeeery long time unfortunately, because it works on a frame by frame basis. I have implemented the solutions proposed by you and it works like I want. But it is not nice that I can't capture more images to RAM.

[dpb]

No, I don't have access to the IMAQ TB other than by downloading a trial and then don't have any hardware to use it with, anyway, if did.

But, I think it's conclusive that Bruno hit the nail on the head; the TB acquisition logic is dumb; it allocates the entire buffer including room for a new image every frame and copies the old buffer over and adds a new frame at the end.

2.2M/frame * 4K frames is the 8.8G; when try to double that by allocating a new buffer + new frame to copy the old over to, it exceeds the 16G you have available and dies. Pretty damning evidence that it occurs at almost exactly the 4K number.

This is a grievous fault in the TB design/implementation; it should keep a list structure or some similar way to implement a growing buffer and only add the additional memory for the next frame. There will be some other overhead associated with keeping that data structure, so won't be able to allocate all memory to an image buffer, but certainly could use a very significant fraction of total memory. Of course, then one will run into issues trying to process those later on if actually do completely fill memory, but...

Unless there's a way to preset the number of images wanted to be collected and that will make it allocate the total memory to begin with instead of just free-running???

If not, I'd say this is a quality of implementation issue deserving of a bug/issue report to TMW and it would appear the only choices one would have to collect more frames to memory would be to add memory to the system.

Alternatively, could you stream to disk instead? If your disk is a SSD, performance shouldn't be too bad compared to memory.