I have a file 1x200 struct how can I distribute it to the worker (32 worker)

Dear all, I have a file 1x200 struct, I would like to send it to the cluster (parallel computation), how can I distribute it to the worker (32 worker). I can make it one by one, but it is really take very long time.

Thanks in advance.

Best regards, ahmed

[Walter Roberson]

Is it a file (text or binary) or is it just a struct()? Do the workers need access to all of it or only to the portion they are acting on?

[Student for ever]

Hello Walter, it just a struct(). I think only to the portion they are acting on. I am sorry I am not familiar with this cluster stuff. Thank you

Answer 1

[Walter Roberson]

Just create the struct normally before the parfor() loop, and inside the parfor() loop, index the struct by the parfor variable. MATLAB will automatically distribute only the needed portion to each worker.

[Ahmed Abdalazeez]

ok, thank you so much... could you explain more please

[Walter Roberson]

data = struct('test', num2cell(rand(10,200),1) );

Now data is a 1 x 200 struct with one field named test, with each entry data(K).test being a 10 x 1 vector. The details of the above are not important: I just needed to create a 1 x 200 struct to show parfor.

Now:

parpool(32)
sums = zeros(1,length(data));
parfor K = 1 : length(data)
   sums(K) = sum( data(K).test );
end

In this example, parfor does not need to copy all of data to every worker: it only copies the portion needed by the worker. Notice that no special code was needed for this: parfor handles it automatically.

This is an example of a "sliced variable", https://www.mathworks.com/help/distcomp/sliced-variable.html

If the use of the elements of the data structure was more complicated, then parfor could decide that the easiest thing to do is to copy all of data to every worker. In that situation, data would be referred to as a "broadcast variable", https://www.mathworks.com/help/distcomp/broadcast-variable.html

[Ahmed Abdalazeez]

Thank you so much Walter