Caenorst opened a new issue #9874: ResNet-50 is slower on Volta since #8302
URL: https://github.com/apache/incubator-mxnet/issues/9874
## Description
I ran the Minimum reproducible example with the setup below at two different version (before
and after #8302):
Here are the results:
d03182f4fc084741ea185aac2eb02d0e7b4151f9 (before #8302):
- real data: 5644 samples / s
- synthetic data: 5971 samples / s
c3e3a832bfeceff070ba263aa8a4489ca27f452e (after #8302):
- real data: 5461 samples / s
- synthetic data: 5740 samples / s
Latest:
- real data: 5425 samples / s
- synthetic data: 5817 samples / s
@ptrendx @DickJC123 @mkolod
## Environment info (Required)
CPUs: Intel Xeon E5-2698 v4 (x2)
GPUs: Nvidia V100 (x8)
## Build info (Required if built from source)
From the default config.mk (in make/config.mk) added:
```USE_CUDA=1
USE_CUDNN=1
CUDA_ARCH := -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode
arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70
USE_CUDA_PATH=/usr/local/cuda
USE_LIBJPEG_TURBO=1
USE_LIBJPEG_TURBO_PATH=/usr
USE_NCCL=1```
## Minimum reproducible example
python /mxnet/example/image-classification/train_imagenet.py --benchmark 0 --gpu 0,1,2,3,4,5,6,7
--batch-size 1024 --num-epochs 1 --data-train /data/imagenet/train-480-val-256-recordio/train.rec
--data-train-idx /data/imagenet/train-480-val-256-recordio/train.idx --data-val /data/imagenet/train-480-val-256-recordio/val.rec
--disp-batches 100 --network resnet-v1 --num-layers 50 --data-nthreads 40 --min-random-scale
0.533 --max-random-shear-ratio 0 --max-random-rotate-angle 0 --max-random-h 0 --max-random-l
0 --max-random-s 0 --dtype float16 --kv-store device
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
|