singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-singa] chrishkchris edited a comment on issue #535: SINGA-490 Optimize performance of stochastic gradient descent (SGD)
Date Thu, 19 Sep 2019 03:02:49 GMT
chrishkchris edited a comment on issue #535: SINGA-490 Optimize performance of stochastic gradient
descent (SGD)
URL: https://github.com/apache/incubator-singa/pull/535#issuecomment-532944885
 
 
   Since I have done some optimization of the framework, I need to retest again the distributed
performance: 
   I tested the distributed training in AWS p2.x8large, after adding the Sync() in the SGD
loop of resnet.py and resnet_dist.py.
   The speed up of using 8 GPUs is now 7.21x, but this is compared without real data feeding.

   See the following throughput comparison in resnet.py and resnet_dist.py:
   
   ```
   ubuntu@ip-172-31-28-231:~/incubator-singa/examples/autograd$ python3 resnet.py
   Start intialization............
   100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|
100/100 [01:23<00:00,  1.19it/s]
   Throughput = 38.13589358185999 per second
   Total=0.8391045022010803, forward=0.26401839971542357, softmax=0.0020227289199829103, backward=0.5730633735656739,
sgd=0.016838366985321044
   
   ubuntu@ip-172-31-28-231:~/incubator-singa/examples/autograd$ /home/ubuntu/mpich-3.3/build/bin/mpiexec
--hostfile host_file python3 resnet_dist.py
   Start intialization...........
   100%|██████████| 100/100 [01:33<00:00,  1.08it/s]
   Throughput = 274.9947180123401 per second
   Total=0.9309269714355469, forward=0.2690380573272705, softmax=0.0021610450744628906, backward=0.6597278690338135,
sgd=0.10374969005584717
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message