mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] Ujjalbuet opened a new issue #9768: Regarding data distribution for multi-GPU implementation
Date Thu, 01 Jan 1970 00:00:00 GMT
Ujjalbuet opened a new issue #9768: Regarding data distribution for multi-GPU implementation

URL: https://github.com/apache/incubator-mxnet/issues/9768
 
 
   As a test I profiled the training of example MLP network with MNIST dataset on 2 GPUs using
nvprof (/usr/local/cuda-9.0/bin/nvprof -o mnist_mlp.nvvp python train_mnist.py --network mlp
 --num-epochs 1 --gpus 0,1 --batch-size 200)
   
   Since the batch size per GPU is 100 and each GPU is supposed to work on half of the dataset,
I expected to see 350 HtoD MemCpys to send half of the dataset to each of the GPU. But the
profiler result shows there are 700 HtoD MemCpys for each of the two GPUs. 
   
   I have the following questions based on the observation:
   1) Does MXNet send the entire dataset to each of the GPUs? If so, why do we need to send
the entire dataset to each of he GPUs? 
   2) If the entire dataset is sent to both the GPUs, does each of the GPU works on entire
dataset or half of the dataset?
   
   It will be helpful if you can explain the details about the input data distribution for
multi-GPUs?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message