mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-mxnet] kpuatamazon commented on issue #17559: [MXNET-1446] Quantization: intgemm matrix multiply wrappers
Date Mon, 24 Feb 2020 14:59:38 GMT
kpuatamazon commented on issue #17559: [MXNET-1446] Quantization: intgemm matrix multiply wrappers

URL: https://github.com/apache/incubator-mxnet/pull/17559#issuecomment-590363395
 
 
   While I agree with the principle that all operators should be parallel (and intend to parallelize
mine), it's important to find the optimal place for parallelism.  
   
   This benchmark shows the right place is over sentences, not inside operators, at least
with Sockeye on current MXNet.  In Marian, we've turned off all OMP threading and just use
sentence-level parallelism.  
   
   Setup: float32 models using MKL and 1 sentence at a time (admittedly small, but common
use case in inference).  Nothing special from this pull request.  
   
   One OMP thread, parallelize across sentences with separate processes.  
   ```bash
   export OMP_NUM_THREADS=1
   time parallel --block 10k --line-buffer --pipe -k python3 -m sockeye.translate --use-cpu
-m model --restrict-lexicon model/lexicon --beam-size 5 <test
   ```
   ```
   real  1m17.884s
   user  37m32.076s
   sys 1m38.002s
   ```
   
   This version loaded Sockeye 41 times!  Each of these takes:
   ```
   real  0m3.528s
   user  0m7.050s
   sys 0m1.240s
   ```
   
   Use all 24 hyperthreads for OMP:
   ```bash
   unset OMP_NUM_THREADS
   python3 -m sockeye.translate --use-cpu -m model --restrict-lexicon model/lexicon --beam-size
5 <test
   ```
   ```
   real  4m26.204s
   user  103m50.951s
   sys 2m10.632s
   ```
   
   This is in line with results reported with Intel, though I just took it further to one
process per core.
    https://raw.githubusercontent.com/awslabs/sockeye/master/docs/tutorials/cpu_process_per_core_translation.py

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message