Communicating between tasks and submitting client in parallel jobs

I've got a long job I've broken into several tasks to run on MS HPC and I am trying to figure out a way to send progress back to the submitting client computer. I'm using createJob/createTask to run the job. Right now I'm polling for the individual task states and this works, but I'd like to get some insight into where each task is in it's progression (e.g., 60% complete). Some ideas I'm condsidering or have tried include:

Using the task objects UserData property, but this appears to only be updated on the matlab worker (for example, one can print it to the Diary just fine from within the function the task is running).

The DataQueue object looks promising but it looks like it's only available when using parallel pools.

I've thought of writing to the diary and then on the calling/client machine having something read these every couple of minutes across the network and present a progress bar for each task. My principle hesitation being this seems a little kludgy.

Any other ideas or suggestions?

Thanks

Answer 1

[Prashanth Ramesh]

While I admit there does not exist a clean documented way of achieving your workflow, I believe implementing parallel pool with a single worker may be a workaround for this issue. Please note that MATLAB

does not add additional overheads between CPU and parallel pool, if the worker size is 1. The only caveat however is that your existing code for the jobFunction should be compatible with "parfeval" .

parpool(1); % Only 1 worker required
Q= parallel.pool.DataQueue;
Q.afterEach(@updatWaitBar);
future = parfeval(@jobFunction, 0, queue);

You could also consider using a similar workflow as mentioned in the below link, although it is implemented for "parfor", the same can be achieved for jobs:

https://www.mathworks.com/matlabcentral/fileexchange/32101-progress-monitor-progress-bar-that-works-with-parfor