mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] xiaodansong opened a new issue #8541: forward() time fluctuate and physical memory gradually increases
Date Thu, 01 Jan 1970 00:00:00 GMT
xiaodansong opened a new issue #8541: forward() time fluctuate and physical memory gradually
increases
URL: https://github.com/apache/incubator-mxnet/issues/8541
 
 
   ## Description
   I am trying to test the prediction time on CPU with one thread of mxnet by calling mod.forward()
function in a loop. However,
   - The time of `mod.forward()` fluctuates a lot.
   - I find that the physical memory (RmRSS) gradually increases with the loop going forward.

   It is a little weird. Can you help me?
   
   ## Environment info (Required)
   ```
   ----------Python Info----------
   ('Version      :', '2.7.6')
   ('Compiler     :', 'GCC 4.8.4')
   ('Build        :', ('default', 'Oct 26 2016 20:30:19'))
   ('Arch         :', ('64bit', 'ELF'))
   ------------Pip Info-----------
   ('Version      :', '9.0.1')
   ('Directory    :', '/usr/local/lib/python2.7/dist-packages/pip')
   ----------MXNet Info-----------
   ('Version      :', '0.12.0')
   ('Directory    :', '/usr/local/lib/python2.7/dist-packages/mxnet')
   ('Commit Hash   :', '4f2af2d2e5216ab3a1faadcc117709b6836029dc')
   ----------System Info----------
   ('Platform     :', 'Linux-3.19.0-25-generic-x86_64-with-Ubuntu-14.04-trusty')
   ('system       :', 'Linux')
   ('node         :', 'xxxxxx')
   ('release      :', '3.19.0-25-generic')
   ('version      :', '#26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015')
   ----------Hardware Info----------
   ('machine      :', 'x86_64')
   ('processor    :', 'x86_64')
   Architecture:          x86_64
   CPU op-mode(s):        32-bit, 64-bit
   Byte Order:            Little Endian
   CPU(s):                56
   On-line CPU(s) list:   0-55
   Thread(s) per core:    2
   Core(s) per socket:    14
   Socket(s):             2
   NUMA node(s):          2
   Vendor ID:             GenuineIntel
   CPU family:            6
   Model:                 79
   Stepping:              1
   CPU MHz:               1200.000
   BogoMIPS:              4790.44
   Virtualization:        VT-x
   L1d cache:             32K
   L1i cache:             32K
   L2 cache:              256K
   L3 cache:              35840K
   NUMA node0 CPU(s):     0-13,28-41
   NUMA node1 CPU(s):     14-27,42-55
   ----------Network Test----------
   Setting timeout: 10
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0044 sec, LOAD: 2.6689
sec.
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0049 sec, LOAD: 0.5542 sec.
   Error open FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz,
<urlopen error [Errno 104] Connection reset by peer>, DNS finished in 0.267659187317
sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0045 sec, LOAD: 1.6930 sec.
   Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0045 sec, LOAD: 1.3875 sec.
   Error open Gluon Tutorial(cn): https://zh.gluon.ai, HTTP Error 401: Authorization Required,
DNS finished in 0.00547504425049 sec.
   ```
   
   Package used (Python/R/Scala/Julia):
   I'm using `Python`
   
   ## Minimum reproducible example
   - Here is my network file (`model-symbol.json`): 
   ```
   {
     "nodes": [
       {
         "op": "null", 
         "name": "data1", 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "data2", 
         "inputs": []
       }, 
       {
         "op": "Concat", 
         "name": "concatData", 
         "attr": {"num_args": "2"}, 
         "inputs": [[0, 0, 0], [1, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv1_weight", 
         "attr": {
           "kernel": "(5, 5)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(2, 2)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv1_bias", 
         "attr": {
           "kernel": "(5, 5)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(2, 2)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv1", 
         "attr": {
           "kernel": "(5, 5)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(2, 2)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[2, 0, 0], [3, 0, 0], [4, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu1", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[5, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv2_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv2_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv2", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[6, 0, 0], [7, 0, 0], [8, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu2", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[9, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv3_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv3_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv3", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[10, 0, 0], [11, 0, 0], [12, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu3", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[13, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv4_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv4_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv4", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[14, 0, 0], [15, 0, 0], [16, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu4", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[17, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv5_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv5_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv5", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[18, 0, 0], [19, 0, 0], [20, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu5", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[21, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv6_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv6_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv6", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[22, 0, 0], [23, 0, 0], [24, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu6", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[25, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv7_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv7_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv7", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "64", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[26, 0, 0], [27, 0, 0], [28, 0, 0]]
       }, 
       {
         "op": "Activation", 
         "name": "relu7", 
         "attr": {"act_type": "relu"}, 
         "inputs": [[29, 0, 0]]
       }, 
       {
         "op": "null", 
         "name": "conv8_weight", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "1", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "null", 
         "name": "conv8_bias", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "1", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": []
       }, 
       {
         "op": "Convolution", 
         "name": "conv8", 
         "attr": {
           "kernel": "(3, 3)", 
           "no_bias": "False", 
           "num_filter": "1", 
           "pad": "(1, 1)", 
           "stride": "(1, 1)"
         }, 
         "inputs": [[30, 0, 0], [31, 0, 0], [32, 0, 0]]
       }, 
       {
         "op": "broadcast_add", 
         "name": "sum", 
         "inputs": [[0, 0, 0], [33, 0, 0]]
       }
     ], 
     "arg_nodes": [
       0, 
       1, 
       3, 
       4, 
       7, 
       8, 
       11, 
       12, 
       15, 
       16, 
       19, 
       20, 
       23, 
       24, 
       27, 
       28, 
       31, 
       32
     ], 
     "node_row_ptr": [
       0, 
       1, 
       2, 
       3, 
       4, 
       5, 
       6, 
       7, 
       8, 
       9, 
       10, 
       11, 
       12, 
       13, 
       14, 
       15, 
       16, 
       17, 
       18, 
       19, 
       20, 
       21, 
       22, 
       23, 
       24, 
       25, 
       26, 
       27, 
       28, 
       29, 
       30, 
       31, 
       32, 
       33, 
       34, 
       35
     ], 
     "heads": [[34, 0, 0]], 
     "attrs": {"mxnet_version": ["int", 1100]}
   }
   ```
   - Here is `test.py` file: 
   ```
   import numpy as np
   import os
   import gzip
   import struct
   import time
   from collections import namedtuple
   import mxnet as mx
   
   wid = 832
   hei = 480
   
   model_name='model'
   sym, arg_params, aux_params = mx.model.load_checkpoint('XXXX', 0)
   mod = mx.mod.Module(symbol=sym, context=mx.cpu(), data_names=['data1', 'data2'], label_names=None)
   mod.bind(data_shapes=[('data1', (1, 1, hei, wid)), ('data2', (1, 1, hei, wid))], for_training=False)
   mod.set_params(arg_params, aux_params, allow_missing=True)
   
   nb_itr = 20
   timings = []
   for i in range(nb_itr):
       data1 =  np.zeros((1, 1, hei, wid), dtype=np.float32)
       data2 =  np.zeros((1, 1, hei, wid), dtype=np.float32)
   
       start_time = time.time()
       mod.forward(Batch([mx.nd.array(data1), mx.nd.array(data2)]))
       tmp = time.time() - start_time;
   
       print('%10s :  %f'% ('mxnet',  tmp))
       #timings.append(time.time() - start_time)
   ```
   - The results are shown below:
   ```
        mxnet :  0.005801
        mxnet :  0.002354
   MKL Build:20170720
        mxnet :  0.767726
        mxnet :  1.809435
        mxnet :  1.526410
        mxnet :  1.474105
        mxnet :  1.604314
        mxnet :  0.611293
        mxnet :  1.718852
        mxnet :  0.746450
        mxnet :  1.469410
        mxnet :  1.606903
        mxnet :  0.612172
        mxnet :  1.722195
        mxnet :  0.747340
        mxnet :  1.469609
        mxnet :  1.605465
        mxnet :  0.611577
        mxnet :  1.721764
        mxnet :  0.746009
   ```
   ## I have tried to set the following `environment variables`:
   ```
   export MKL_NUM_THREADS=1
   export OMP_NUM_THREADS=1
   export OPENBLAS_NUM_THREADS=1
   export MXNET_CPU_WORKER_NTHREADS=1
   export MXNET_CPU_PRIORITY_NTHREADS=1
   export MXNET_CPU_NNPACK_NTHREADS=1
   export MXNET_KVSTORE_REDUCTION_NTHREADS=1
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message