GPUを使用したDe​epLearning​の並列処理に関する質​問

現在，GPU（GeForce 1080-Ti）を使用したCNNによる解析を行っています．1枚のGPUで動作させていますが，高速処理のため，複数枚数（2枚以上）のGPUを考えています．どのような仕様（また，環境下）であれば，CNNの複数GPUを用いた処理が可能でしょうか．以下の環境下で動かしたいと考えています．

・Windows10環境下 ・GPU（GeForce 1080-Ti）×2枚 ・Parallel Computing Toolboxは利用可能

Answer 1

[michio]

具体的な実行方法としては下記が参考になるかと思います。

• 畳み込みニューラルネットワークの計算に複数の GPU を使用する方法はありますか？
• R2017a で R-CNN, fast/faster R-CNN の計算を複数の GPU 上で行うことは可能ですか？

[Joss Knight]

Are you asking me a question? I can't tell. If so, the answers are:

• Yes, multi-GPU training is supported, just look for the ExecutionEnvironment training option.
• No, R-CNN does not support multi-GPU or parallel training.

[michio]

I apologize for the confusion, Joss.

The two bullets are the links to MATLAB Answers posts whose titles are in the form of questions. I posted these because I thought they could potentially answer Nagai-san's question.

[Joss Knight]

Many apologies, I should have noticed they were links! (They weren't links in Google translate of course...)

[michio]

No worries :)

Answer 2

[Joss Knight]

MATLAB CNNs support any modern multi-gpu environment and it's difficult to advise. Amazon and Azure provide cloud environments of 8 and 16 Tesla K80 GPUs, while the typical 'multi-gpu system in a box' retail system would allow you to put together 4 or 8 GeForce GPUs (1080 or Titan X) and that would be equally useful for Deep Learning. If you just want to try it out, put another 1080i in your desktop, if your PCI bus can take another.

There are two bits of advice I can usefully give. Firstly, avoid setting up your multi-gpu cluster under Windows, because MATLAB's cluster model is multi-process not multi-threaded. Multi-process scales better to multiple machines, but it prevents peer-to-peer communication between GPUs on Windows, since P2P inter-process communication in CUDA is only supported on Linux (for now). It's worth pointing out that you can still use MATLAB under Windows as your client while the cluster runs under Linux.

Secondly, try to put as many GPUs as possible on the PCI bus of one machine, rather than dividing them between multiple machines. Again, this is because of peer-to-peer communication, which is essential for fast training of deep networks. Communicating between nodes on a multi-node cluster is much slower than peer communication between GPUs on the same PCI bus.