Sharing Information Between Client and Workers with the Parallel Computing Toolbox

Hello, I have started working with the Parallel Computing Toolbox and have three questions:

1. The first time (in a Matlab session) I try to create a job, Matlab hangs for a very long time (maybe 4 minutes or so). The subsequent createJob executions are much faster. This is also the case when trying to "validate" the cluster profile. Matlab gets stuck on the createJob task. Why is that?

2. Is is possible to share objects between the Matlab client and workers? I would like the worker to update an object's properties that the client would then be able to read. An example would be to update the status of an object as the worker executes a function (eg: phase one complete, phase 2 complete...). I have tried using the jobData property of the MATLAB job to store the object, but its properties do not get updated...

3. This is similar to two in that it is about sharing information between the client and the worker during execution: Is it possible to define custom callbacks during a worker's execution. My worker could thus send back partial data as soon as it finished computing it by executing this callback. The callback would be caught by the client and displayed in a plot (for example).

Thanks, Jeremy

Answer 1

[Jeremy]

I am answering my own question after speaking to Matlab support so that it may be useful to others:

1. While I was recommended to disable the local MPI using

distcomp.feature('LocalUseMpiexec', false)

my problem was actually caused by a LM_LICENSE_FILE environment variable (source, this StackExchange. Deleting the variable fixes the problem, but this will of course prevent other software that depend on it from running.

2. There seem to be no way to exchange information between the client matlab and a worker, except of course at the beginning and at the end of the execution of the job. From Matlab support:

"All the workers will have their local copies of the variables defined in the client. If you want to send handle objects to the body of a parfor-loop, you need to explicitly assign the modified handle objects to the output variables of the parfor-loop."

3. From Matlab support:

"I am afraid that currently there is no way to define a callback function during a worker's execution. A possible workaround is that you can use a timer event callback on the client that will be fired at some regular intervals and the callback keeps polling the status of the job. The state of a job can be one of the following: pending, queued, running, finished, and failed."

[Edric Ellis]

Could I ask: was your LM_LICENSE_FILE variable large, or did it reference files that don't exist?

[Jeremy]

Hi Edric,

My LM_LICENSE_FILE variable contains a single entry: a reference to a local port on my machine. I would be wiling to help debug this issue, as it would be very inconvenient to have to delete and remake the variable each time I need either software.

If this helps, the function that hangs is +apishared\private\getJobInitData. On my machine, its self-time is about 174 seconds.

[Edric Ellis]

Can I suggest you contact MathWorks technical support about this - they would almost certainly be able to help better than I can.

Answer 2

[Edric Ellis]

If you want to perform interactive parallel computing, you might be better off investigating the capabilities of the parallel pool. This connects up several MATLAB workers to your client MATLAB, and allows them to be used interactively for parfor loops, smpd blocks and parfeval calls. With parfeval, you might be able to do something like you're after - it lets you invoke functions on the workers sending data and getting back output arguments - without the overhead of launching a separate job (the workers are already running as part of the parallel pool).

There's an entry on Loren's blog here that shows how you can use parfeval to achieve interactive parallelism.

[Jeremy]

Hey Edric, thanks for your suggestion. The goal I am trying to achieve is to control instruments (TCP/IP or USB with the Digital Acquisition Toolbox) to perform an experiment. Since that experiment will take a long time to run, I would like to run it in the background such that the main matlab process may do other things.

I tried just wrapping the function that runs the experiment into a parfeval, but ran into problems with non-serializable objects (in that case, it was due to a database function, as I can't try it with an actual instrument right now; however, TCP objects are apparently also non-serializable).

Do you know if what I am trying to do is possible?

[Edric Ellis]

For this sort of application, you'll need to set up the objects within the function that you call within parfeval. If you need to set up an object and then use it multiple times, you can use the Worker Object Wrapper together with parfeval.