Script to run a Matlab Script on the event of crash

I run a matlab script (call it INIT.m) that takes care of parallel toolbox initialization and running original script. Now, sometimes, because of some unpredictable problem of data, parallel toolbox stops due to error in matlabpool job (possibly).

I would like to re-run INIT.m on the event of this kind of crash.

It can be another matlab script or may be a shell script/java code.

Any sort of help will be appreciated.

Thanks in advance.

##### Update (Error statements as also given in the comments)

It gives following error message and stops (But do not exit)

MATLAB is exiting because of fatal error
??? Error using ==> parallel_function at 598
The session that parfor is using has shut down

It also says that

??? The client lost connection to lab 117.
This  might be due to network problems, or the interactive matlabpool job might have errored

Environment : Linux

[Daniel Shub]

Windows, Mac, or Linux? On Linux you could call MATLAB from the shell and examine the exit code.

[Sarthak]

Yes, I thought of that. But I am not sure about the interfacing. To be exact, how shall I know that the error occured in one of the labs and the entire process is stopped?

[Edric Ellis]

Does MATLAB produce some sort of crash dump when this happens? If so, please contact technical support. It is not expected that a worker crashing should cause your desktop MATLAB to crash.

[Sarthak]

No, it doesn't degenrate crash dump. As I mentioned, matlab doesn't exit. Due to segmentation fault in MEX computation, one worker crashes and as a result the entire process shuts down.

It would be helpful if I can automatically restart the process. If it sends some signal to system, I can trap. But again I am not sure about the interfacing.

Answer 1

[Jan]

What about running a TIMER, which checks every second if all labs are still active?

[Sarthak]

Well, thanks. I need to check how to perform it, i.e. how to get the information about the labs. I am not sure about interfacing. Can you help me with that?

[Daniel Shub]

Assuming that the error does not kill the timer, you could probably check lasterror

[Sarthak]

For once, it works when I keep my initial script (INIT.m) as the callback function. But next time onwards, as the callback function, i.e. INIT.m encounters seg fault, timer stops.

Anyway, this idea works for one re-run.

I guess, I may have to find some workarounds for multiple runs.

Any solution/tips will be appreciated. :)

Answer 2

[Ahmed]

The "try .. catch .. end" construct might be what you are looking for. In case an error occurs between try and catch, then the code portion between catch and end will be executed. However, this works if there is an error, if Matlab literally crashes, it won't work.

   doc try

[Sarthak]

try catch won't work in this case as you mentioned. It's a Matlab crash.. Can you tell any other way out?

[Matt Kindig]

I'm curious whether Matlab calls finish.m when it crashes. You could try to put your crash logic in a custom finish.m script.

doc finish

That might work...

[Sarthak]

I am working on parallel environment. So, if problem occurs in one of the connect6ed labs, entire process is shut down. But matlab does not exit; the process rather stops saying that

MATLAB is exiting because of fatal error
??? Error using ==> parallel_function at 598
The session that parfor is using has shut down

It also says that

??? The client lost connection to lab 117.
This  might be due to network problems, or the interactive matlabpool job might have errored

And finish.m is invoked when we exit from matlab. So, I guess, in this case, it won't be invoked.