Log file of minimization done in parallel using batch

Hi,

I'm working on a problem that has an outer minimization that searches over one set of parameters, and an inner minimization that for every iteration of the outer one, solves n systems of equations, where each system depends on the parameter values of the iteration of the outer loop. As all systems of the inner loop are independent of each other, I can use the parallel features of MATLAB to speed up the outer minimization. This work perfectly when using the 'local' profile in my laptop and I save a lot of time by using PARFOR in the inner minimization, compared with the same situation without parallelizing the inner loop.

However, I need to submit this as a batch job to a remote cluster, and I would like to be able to monitor the job. In particular, I would like to see the value of the objective function at each iteration, as it is shown in my laptop when 'Display','iter' is specified in options.

I have tried using

job = batch('script', 'matlabpool', 4,'CaptureDiary',true)

However, using

diary(job)

while the job is running returns nothing. It seems that I'll have to wait to see the output of diary() until the job is finished. Is there any way to observe what I want? I would like to have a log file as the one that is generated when I run the script as a batch job without using the parallel features.

Thanks,

Fernando

Answer 1

[Edric Ellis]

At this time, it's not possible to get the diary output from a batch job until after the job has completed execution. You could manually write status output to a file using fopen/fprintf/fclose.

EDIT

Recent versions of Parallel Computing Toolbox do support incremental updates to the diary of a batch job. For example:

c = parcluster('local');
j = batch(c, 'for idx = 1:5, disp(idx), pause(2), end');
% Wait for the job to start running.
wait(j, 'running');
% Print the diary in a loop.
while ~strcmp(j.State, 'finished')
    fprintf('Diary at time: %s\n', string(datetime));
    diary(j);
    pause(2);
end

Results in output like this:

Diary at time: 24-Nov-2020 13:57:01

Warning: The diary of this batch job might be incomplete because the job is still running.

--- Start Diary ---

1

2

3

--- End Diary ---

[Fernando]

Hi Edric,

Thanks for your answer,

Fernando