how parfor distributes resources to each iteration if the number of cores is larger than the number of iterations

Here is the situation: I want to perform a few iterations using parfor on a cluster where I can get access to more but finite cores than the number of iterations. Given that each iteration consumes very much time and memory, say the total memory of 10 cores, but the average number of cores allocated for each iteration is less than 10, how should i configure the parfor to avoid the out-of-memory error? Does the parfor distributes resources to each iteration evenly?

Answer 1

[Damian Pietrus]

Since you mentioned working on a cluster, the first thing I'd like to address is making sure that you are requesting enough resources from the scheduler. If you are using the default values or requesting too little memory, this could be the cause of the out of memory errors you are seeing rather than the way parfor distributes iterations.

For example, if using the Slurm scheduler, you can use the --mem-per-cpu flag to request a certain amount of memory per core for your job. As an example, if you request 10 GB per cpu and use 10 workers in your parpool, then you will request a total of 100GB of memory on the compute node. (10GB x 10 cpu = 100 GB total). Increase the memory request as needed until your job has the memory it needs.

You may need to take a look at the documentation for your cluster to see what types of nodes are available and how much memory each node has. Often clusters have "high mem" nodes in addition to standard nodes.

Give that a shot first and let me know how it goes.

[XYC]

Thanks very much for your reply. Before I post this question, i did some tests using the default mem-per-cpu. Let's say I have 5 iterations (each iteration needs 21GB memory) to perform using parfor, and I request 16 cores (each core has 4GB). It is OK if I request 6 cores (total 24GB>21GB) to perform each iteration sequentially, but performing 5 iterations using parfor with all 16 cores invokes out-of-memory error because parfor will perform all of them simultaneously (total 64GB requested < total 105GB needed).

I wonder if there is a way to tell parfor to perform first three iterations out of the total 5 simultaneously and then the rest two. Or can i somehow specify the number of cores distributed to each iteration and thus prevent the parfor performs them simultaneously.

All in all, I will give it a try first to increase the memory per cpu.

[XYC]

I try to change the --mem-per-cpu flag but got the following error:

sbatch: error: Batch job submission failed: Job submission failed because too much memory was requested relative to the number of CPUs requested. The requested memory:CPU should be kept no more than DefMemPerCPU.

It seems I have no choice but to request more cores.

Answer 2

[Edric Ellis]

When a parfor loop executes on a parallel pool, PCT will divide up the entire loop range 1:N into a series of batches known as "sub-ranges". This division depends on the total number of iterates in the full range, and the number of workers available. By default, PCT tries to send batches that are big enough that communication costs are minimised, but also small enough that in the case where different iterations take different amounts of time, the workers are kept as busy as possible.

You can override this default division by using parforOptions. For example, to force parfor to send iterates individually, you can do this:

pfo = parforOptions(gcp(), RangePartitionMethod="fixed", SubrangeSize=1);
parfor (i = 1:10, pfo)
    out(i) = feature('getpid');
end
out'

[XYC]

Thanks for replying! As far as i can understand, setting SubrangeSize=1 will force parfor to send one iteration to each worker in the case where the num of iterations is larger than that of workers. What would happen if there are more cores (or workers) than the iterations, and are there some workers remain idle? What i want is combine several cores to form a worker for each time- and memory-consuming iteration and i have no clue how to do it. And i use parfor to automatically start a parpool without specifying the num of workers in a cluster with many cores.

[Edric Ellis]

Yes, with that scheme, if there are more workers than iterations, you will not keep them all busy. (Sorry, somehow I overlooked that part of your question - I was mostly answering to clarify exactly how the iteration batching works).

I think the solution here is going to involve working with your cluster configuration to ensure the workers you get each have enough memory to run an iteration, rather than anything related to parfor specifically.

One way to head towards what I think you need is to use multi-threaded workers. From the MATLAB client side, you specify the NumThreads property in your parallel.Cluster profile. This needs support from your cluster integration scripts (I'm not an expert here I'm afraid). The result though is that you can end up with a parallel pool where each worker process is multi-threaded. This should have the desired side-effect that each worker process has more memory available. Also, if your time-consuming function happens to take advantage of MATLAB's intrinsic multithreading (e.g. large matrix operations), then that will also be a benefit.

[XYC]

thanks again for the answer! I'll look through the docs for more details about multi-threaded workers.

Answer 3

[Matt J]

You have to consider the number of workers, M, not just the number of cores, C.

If M is the number of parpool workers and N is the number loop iterations, then each worker will be assigned a consecutive subsequence of N/M iterations, which it will run serially. So, the amount of memory each worker will try to consume is the amount of memory used by N/M of your loop iterations, whatever that is.

The amount of memory each worker has available to it is RAMTotal/M.

Obviously, this math becomes slightly more complicated if N/M or C/M are not integers. Maybe also if your cores are tied up by other Apps besides Matlab.

[XYC]

Thanks, Matt. I think you are right. However, in my case, there are more cores (or workers?) than the number of iterations, and i try to figure out how the resources, especially memory or RAMTotal as you say, are distributed among the iterations so that each iteration can have enough memory to finish.

[Matt J]

I don't think it makes sense to create more workers than iterations. If you create fewer workers so that M=N, then each iteration will be assigned to 1 worker, and should have RAMTotal/N GB available to it.

[Walter Roberson]

If M is the number of parpool workers and N is the number loop iterations, then each worker will be assigned a consecutive subsequence of N/M iterations, which it will run serially.

It is more complicated than that.

When M << N (workers less than iterations) then first roughly 2/3 of N is divided up into M chunks and queued. Then roughly 1/4 is divided up into M chunks and queued; then the remaining iterations are queued individually. This allocation process is used to account for the possibility that some of the workers might happen to complete notably faster than the others -- there will still be meaningful chunks for the second round.

[XYC]

@Walter Roberson Hi, Walter, thanks for comment! Do you know what would happen if the available cores are many more than the number of loop iterations? In my case each iteration is time- and memory-consuming and I have to request many cores to ensure enough memory. I use parfor function to automatically start a parpool. I wonder how many workers are invoked automatically and if all the cores are occupied.

[Matt J]

I wonder if there is a way to tell parfor to perform first three iterations out of the total 5 simultaneously and then the rest two

You could break it up into 2 loops,

parfor (i=1:3,M)
    ...
end

parfor (i=4:5,M)
    
end

[XYC]

it's a good idea but i have a long loop body, so i have to convert the loop body into a function that can be called inside both the parfor loops.

[Walter Roberson]

Sorry, I do not have any information about how resources are allocated for distributed computing.

[XYC]

@Walter Roberson It's all right. Thanks anyway!