Coordinate workers in parallel generic algorithm optimization

Hi,

I am doing an optimization using the generic algorithm. It is a very long optimization and I am using the parallel calculation.

The cost function is long above does multiple things. To simplify, let's say that does the following: 1.-calculates a data set 2.-gets a sample 3.-calculates the cost

During step one (data set calculation) it uses a lot of ram, and ram is the bottleneck.

During step three (cost calculation), this is no longer the case, given that I am using a sample.

I can not use the sample to start with, unfortunately (before taking a sample I need to calculate running averages that depend on the last n points).

I have 6 cores available. During step 1 I can use at most 3 (to avoid running out of memory). During step 3 I can use all 6.

Is there a way to coordinate the 6 workers during execution?

I have tried using final calculates, keeping a count of the number of workers in each step and using the pause function for a worker of the number of workers in the next step is to high. For some reason this does not work, perhaps the global variables are only updated once the execution of the cost function terminates, but I need the global variables to be updated several times during the execution of the cost function.

Any ideas on how to solve?

[Joe]

Instead of "I have tried using final calculates" I wanted to write "I have tried using global variables "

[Grant]

Have you considered using parfeval? It allows asynch calculations in parallel and you should only assign one worker per function call.

The data you calculate will be 100% separated from each other so what you could do is have it run a fixed number of iterations then gather the data, or you could use synchronized function calls so that it will be able to gather the data at the same time if its really important.

I will say that parfeval() allows you to gather data as it becomes available.

http://www.mathworks.com/help/distcomp/parfeval.html

[Joe]

parfeval is a great learning from today, thanks for letting me know! I don't think I can use it for this particular case. I should program the generic algorithm myself, and I really want to avoid that.

What I ended up doing was doing the calculation with 3 cores...