Using Parallel Computing Toolbox to run code on CPU and GPU simultaneously

Hi all, I`m running an optimization code that even with Parallel Computing Toolbox on the CPU lasts several days.

I have a i9 CPU and 2 Nvidia RTX A5000 GPUs.

Is there a posibility to run the code simultaneously on the CPU and GPUs?

Please help!

[Raymond Norris]

Can you provide some more context here?

• How long does it take to run the code you want to speed up to run serially? If you're entire body of code runs in 2 weeks, there may be only part of the code that can be improved.
• Have you addressed code analyzer tips, like preallocation?
• Is it possible you're offsetting implicit multi-threading (e.g., fft) with explicit multi-processing (e.g., parfor) and getting a zero-sum gain on a single machine?
• How are you using the Parallel Computing Toolbox (explicitly calling parfor, parallel-enabled functions, etc.)?

[Walter Roberson]

To clarify, it is not possible to just to something like

 parfeval(GpuDevice2, @handle_to_function, parameters)

in order to run m code on a gpu.

Functions that are defined for gpu use run on gpu if the function is invoked with a gpu array parameter. So you have to construct arrays on the gpu and run functions on those arrays. You would only want to do this on workers that have a gpu allocated to them. So basically you would need two variations of the code (or careful writing of a single version) with one version tuned to run on gpu and the other that does no gpu work at all. You would have to keep track of which thread was of which type.

It is not difficult to do this, just not as simple as saying "run this on gpu"

Answer 1

[Walter Roberson]

%outline
max_available_gpu = gpuDeviceCount();
max_available_workers = 8;
num_available_gpus = max_available_gpu;
num_available_workers = max_available_workers;
available_gpu_list = 1 : max_available_gpu;

number_of_outputs = 2; %for this example

while there are still tasks to do
    if num_available_gpu > 0
        which_gpu = available_gpu_list(1);
        available_gpu_list(1) = [];
        num_available_gpu = num_available_gpu - 1;
        run_something = true;
    elseif num_available_works > 0
        which_gpu = [];
        MaybeOnGPU = @(EXPRESSION) EXPRESSION;  %instead of creating it on GPU
        run_something = true;
    else
        run_something = false;
    end
    if run_something
        future_index = future_index + 1;

        futures(future_index) = parfeval(@task_to_execute, number_of_outputs, which_gpuappropriate parameters);
    else
        [used_gpu_number, results{1:number_of_outputs-1}] = fetchNext(futures);
        if ~isempty(used_gpu_number)
            available_gpu_list(end+1) = used_gpu_number;
            num_available_gpu = num_available_gpu + 1;
        end
        do something with the results
    end
end

function [which_gpu, result1] = task_to_execute(which_gpu, appropriate parameters)
   if isempty(which_gpu)
       MaybeOnGPU = @(EXPRESSION) EXPRESSION;  %return expression unchanged
   else
       gpuDevice(which_gpu);
       MaybeOnGPU = @gpuArray;
   end
   
   %stuff here
   FirstVariable = MaybeOnGPU(ones(5,8));
   SecondVariable = MaybeOnGPU(zeros(8, 4));
   result1 = FirstVariable * SecondVariable;
   
   if ~isempty(which_gpu)
       [result1] = gather(result1);
   end
end

This code keeps track of available GPU. When a GPU is available, the GPU number is passed in to the function; otherwise empty is passed there. The function looks to see if the GPU number is empty to decide whether variables should be put on the GPU; if so a wrapper is created that punts data to GPU, and otherwise the wrapper does nothing. Then use the wrapper as appropriate to potentially place data on the GPU. Then do calculations; the calculations will take place on GPU if any of the data is on GPU. At the end, if we were using GPU, gather() the results.