Parallel computing memory allocation
28 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
I am running code on an HPCC. I have a very large loop so I am trying to split it up. The problem is that I seem to have to give each parallel instance the entire memory reserve for the job leading to too much memory usage.
ClusterInfo.setMemUsage('10gb'); %command from our hpcc
parpool(100)
inside_code_file
Where inside_code_file generates a very large matrix [just under 10 gb] and runs 'parfor' through each cell independently.
When doing it this way, I have to give Matlab 10gb for each of the 100 parallel cores. I'm pretty sure I set up the parfor correctly as there are no `split' warnings.
Is this something I can fix through Matlab or is it on the HPCC side?
Or is this just a fundamental aspect of parallel computing and unavoidable?
0 comentarios
Respuestas (1)
Tushar Upadhyay
el 16 de Jun. de 2017
From the information you provided, It seems that you want to find a better memory efficient way to avoid allocating 10gb of memory to each worker.
Does each worker require a separate copy of the matrix to do their tasks or they can share the same matrix?
2 comentarios
Nagarjuna Manchineni
el 20 de Jun. de 2017
If the input matrix can be shared, then MATLAB will distribute the array to the respective workers during the runtime and once the run for that particular input is completed, then MATLAB worker returns the data back to client (MATLAB).
So, once verify which kind of variable (sliced, broadcast, temporary, loop, etc..) you are trying to use in 'parfor' and let us know.
Also, see the following documentation link for more troubleshooting steps on variables that are shared across workers from MATLAB:
Ver también
Categorías
Más información sobre Parallel Computing Fundamentals en Help Center y File Exchange.
Productos
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!