Running two functions in parallel of which one would like to use parfor

Challenge: I repetitively read external data needing 250ms per data set. Then I process the data which takes 300ms on 20 phyical cores.

Now, it would of course be good to read data, and while the processing is in action, already read the next set. But, since I already use "parfor" in data processing, I cannot put both functions (reading and processing) in an smpd environment.

Ok, I think I know the long answer to this question: Put everything in smpd, send one of the workers in the data reading function and 20 of them to data processing and "rebuild" parfor functionality (means distribute loop indices and accumulate results on a master worker).

Maybe there is a shortcut I don't know? This problem should not be so unusual that there is no chance for an easy solution

Thanks,

Michael

Answer 1

[Mohammad Abouali]

Use non-blocking parfeval().

[Michael Krauß]

Thank you!

[Mohammad Abouali]

you are welcome