Borrar filtros
Borrar filtros

Titan V using huge amount of memory

1 visualización (últimos 30 días)
arvid Martens
arvid Martens el 25 de En. de 2018
Respondida: Joss Knight el 22 de Feb. de 2018
Recently I added a second GPU (titan V) to my workstation. When I try to perform my simulations which use a parpool with each worker acces to the GPU's in the system I noticed that the Titan V uses a lot more memory than my other GPU (Tesla K40c) which results in out of memory and invalid object handles errors. The Titan V uses for the initialization of matlab on the GPU around 514MB whilst the tesla only takes 126 MB on the workers.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.65 Driver Version: 390.65 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
===============================+======================+======================
| 0 TITAN V TCC | 00000000:04:00.0 Off | N/A |
| 33% 48C P8 30W / 250W | 709MiB / 12186MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro K6000 TCC | 00000000:A1:00.0 Off | Off |
| 32% 57C P8 25W / 225W | 459MiB / 12214MiB | 0% Default | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory GPU PID Type Process name Usage | =============================================================================
| 0 14600 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 514MiB |
| 1 4436 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 126MiB |
| 1 24908 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 126MiB |
+-----------------------------------------------------------------------------+

Respuestas (2)

Joss Knight
Joss Knight el 22 de Feb. de 2018
To avoid this cost, do not use the same GPU on multiple workers. Although the GPU is a data parallel system, it is not task parallel - in other words, anything you run on the GPU from different workers will run in serial, gaining you no benefit of overlap. In addition you are experiencing the additional problem that every new CUDA context being opened on the same GPU is causing this extra memory overhead.
To solve this issue, open your parpool with gpuDeviceCount workers, and no more.

Joss Knight
Joss Knight el 8 de Feb. de 2018
Editada: Joss Knight el 8 de Feb. de 2018
The CUDA runtime in the latest drivers appears to be allocating a huge amount of main memory when it is initialized on a Volta card. We are following this up with NVIDIA.
  2 comentarios
arvid Martens
arvid Martens el 20 de Feb. de 2018
Is there any new on the driver issue? It is really hampering my workflow.
Another strange thing is that when I load an instance of matlab using only the tesla K40c it occupies a significant amount (300-400MB) of memory on the titan V even if it is not being used.
Joss Knight
Joss Knight el 22 de Feb. de 2018
NVIDIA have indicated to us that this memory allocation size is expected, holding kernels and page tables. But we are still chasing them further.
The allocation on the Titan V occurs when the CUDA runtime is initialized, which happens when the device is selected. To avoid this, select your second device as the first GPU command you execute, i.e. gpuDevice(2).

Iniciar sesión para comentar.

Categorías

Más información sobre GPU Computing in MATLAB en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by