- Your network is too small and there isn't enough work for the GPU to do, so the time is dominated in other activity such as loading and preprocessing input data, plotting, or sending data between workers.
- Your input data is taking a long time to load. It may be that trainData is huge and is being stored in virtual memory, so indexing into it is taking a huge amount of time. Very large data shouldn't be stored in memory, it should be stored in files and accessed using a Datastore.
- Some hardware issue means you're not actually using all the GPUs.
Cannot utilize fully all GPUs during network training
10 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
Tomer Nahshon
el 22 de Dic. de 2019
Comentada: Joss Knight
el 17 de Feb. de 2020
It’s the performance and use of the resources installed on the Computer (Amazon Cloud EC2 instance in our case).
I am using a p3.8xlarge instance in EC2 on awamzon web server – basically it means I am using 4 GPUs V100,
I am training a neural network.
using:
mdl(i).Net = trainNetwork(trainData(:, :, :, 1: itStep: end), trainLabels(1: itStep: end, :), layers, options);
in options I define 'multi-gpu'
I also defined 'parallel' and tried to play with number of workers but all I see is just more processes waiting in the GPU queue on nvidia-smi.
For some reason I see that all GPU are working (see GPU.png) but for limited amount of time (very high usage for 3 seconds and then drops to 0% for 10 seconds at least.
I looked at the htop information(htop.jpg), I see that not all threads of the CPU are in use so that is the bottleneck (I think?)
I have a xeon processor on this instance with 32 cores (16 physical, 32 logical)
When I try to utilize all threads through local profile (profile local pool.png) it seems like it still doesn’t respond .
I get more workers because of it (CPU ?), but the GPUs still doesn't seem to improve
Tried to increase batch size - but at some point the GPU is out of memory, so that's not the problem.
How do i utilize all cores of the CPU to transfer data to the GPUs?
I read somewhere that you can also load the data to the pool itself? will that help?
I use the https://ngc.nvidia.com/catalog/containers/partners:matlab/tagsmatlab container for matlab:r2019a
I scanned these already:
Would appreciate your help.
Tomer
9 comentarios
Joss Knight
el 29 de En. de 2020
Editada: Joss Knight
el 2 de Feb. de 2020
That will work or just implement an ordinary datastore with the Partitionable mixin, which is the currently advised approach. All you have to do is implement read() to read the data and the targets that go with it, and return a cell array containing both.
"For networks with a single input, the table or cell array returned by the datastore has two columns that specify the network inputs and expected responses, respectively.
For networks with multiple inputs, the datastore must be a combined or transformed datastore that returns a cell array with (numInputs+1) columns containing the predictors and the responses, where numInputs is the number of network inputs and numResponses is the number of responses. For i less than or equal to numInputs, the ith element of the cell array corresponds to the input layers.InputNames(i), where layers is the layer graph defining the network architecture. The last column of the cell array corresponds to the responses."
Respuesta aceptada
Tomer Nahshon
el 2 de Feb. de 2020
Editada: Tomer Nahshon
el 2 de Feb. de 2020
3 comentarios
Joss Knight
el 17 de Feb. de 2020
It could be a number of reasons. With a custom ReadFcn there's no prefetching, so that will limit CPU core usage to one per MATLAB. But also you have 4 GPUs and if they become the compute bottleneck, your CPU cores will be waiting regardless.
You ought to be able to get all the CPU cores working with the DispatchInBackground training option. But that would preclude you using all the GPUs as well. Ideally you would use both DispatchInBackground and multi-gpu training, but I don't think that will work with your custom datastore. To get that going you're going to need to use the MiniBatchable mixin and PartitionableByIndex - because this feature needs to be able to divide your data by index.
Más respuestas (0)
Ver también
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!