mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinbo Ci <>
Subject [apache/incubator-mxnet] [RFC] Use TVMOp with GPU & Build without in CI (#18716)
Date Wed, 15 Jul 2020 06:51:35 GMT
## Problem 1: TVMOp doesn't work well with GPU builds [#17840](

### The error message:

>>> import mxnet as mx
>>> x =[[0, 1], [1, 1], [2, 2]], ctx=mx.gpu())
>>> idx = x < 2
>>> x[idx]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/", line 1013, in __lt__
    return less(self, other)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/", line 8672, in less
    return _mx_nd_np.less(x1, x2, out)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/ndarray/numpy/", line 6869, in less
    return _api_internal.less(x1, x2, out)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/_ffi/_ctypes/", line 115, in
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "../3rdparty/tvm/src/runtime/", line 125
  File "../3rdparty/tvm/src/runtime/", line 94
TVMError: Check failed: ret == 0 (-1 vs. 0) : Check failed: f != nullptr: Cannot find function
less_scalar_gpufloat32_2bool_2_kernel0 in the imported modules or global registry

### Root cause:

In [mxnet/contrib/tvmop/](,
only `function_binary` `(llvm Module)`is saved in ``. The `imported_modules` `(cuda
Module)`is not saved. So TVM cannot import any gpu functions and cannot find `less_scalar_gpufloat32_2bool_2_kernel0.`

(Pdb) func_binary
Module(llvm, 55d7ce519d48)
(Pdb) func_binary.imported_modules[0]
Module(cuda, 55d7c7a09818)

### Solution ([Github PR](

* Save `imported_modules[0]` to `libtvmop.cubin:`
* Define `Import` function (using `TVMOpModule->Import` ):
* Import `cubin_module` to `global_module`:
* Outputs:

>>> import mxnet as mx
>>> x =[[0, 1], [1, 1], [2, 2]], ctx=mx.gpu())
[10:19:41] ../src/ cuDNN lib mismatch: linked-against version 7605 != compiled-against
version 7501. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
[10:19:41] ../src/ Upgrade advisory: this mxnet has been built against cuDNN lib
version 7501, which is older than the oldest version tested by CI (7600). Set MXNET_CUDNN_LIB_CHECKING=0
to quiet this warning.
>>> idx = x < 2
>>> x[idx]
array([0., 1., 1., 1.], ctx=gpu(0))

## Problem 2: CI Checks: does exist on the machine builds mxnet

### The error message:

When running [unix-gpu](

[2020-06-18T08:26:18.355Z] Traceback (most recent call last):
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/contrib/tvmop/", line 20, in <module>
[2020-06-18T08:26:18.355Z]     import tvm
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/3rdparty/tvm/python/tvm/", line
27, in <module>
[2020-06-18T08:26:18.355Z]     from . import tensor
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/3rdparty/tvm/python/tvm/", line 20,
in <module>
[2020-06-18T08:26:18.355Z]     from ._ffi.object import Object, register_object, ObjectGeneric,
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/", line
24, in <module>
[2020-06-18T08:26:18.355Z]     from .base import _FFI_MODE, _RUNTIME_ONLY, check_call, _LIB,
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/", line
65, in <module>
[2020-06-18T08:26:18.355Z]     _LIB, _LIB_NAME = _load_lib()
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/", line
57, in _load_lib
[2020-06-18T08:26:18.355Z]     lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_GLOBAL)
[2020-06-18T08:26:18.355Z]   File "/usr/lib/python3.6/ctypes/", line 348, in __init__
[2020-06-18T08:26:18.355Z]     self._handle = _dlopen(self._name, mode)
[2020-06-18T08:26:18.355Z] OSError: cannot open shared object file: No such
file or directory

### Root cause:

The unix-gpu machine that **builds** mxnet does not have

### Solution 1:

Link `` with `stub/` on the machine that **builds** CI Checks.

### Solution 1 Pros/Cons/Workloads:

* **Pros**: Solve the issue easily.
* **Cons**: Violates the effort of removing `` totally, (would be great if someone
can elaborate the motivation behind it).
* **Workloads**: ~1 week

### Solution 2 (Possible) ([Github PR]( :

TVM links because it invokes CUDA driver API during runtime. While these functions
are not executed during compile-time. Therefore it is possible to remove them for compile-only
I have made a prototype to remove the linkage of from

* Set `target_link_libraries` of tvm and tvm_runtime differently. (CMakeLists.txt)
* Set an variable `CUDA_COMPILE_ONLY` to be `ON` to indicate “Building without” (CMakeLists.txt)
* When `CUDA_COMPILE_ONLY` is `ON`, add compilation definition `-DCUDA_COMPILE_ONLY` (CMakeLists.txt)
* When `CUDA_COMPILE_ONLY` is defined (when compiling, ignore any cuXXX CUDA Driver
API functions: (cmake/modules/CUDA.cmake, src/runtime/cuda/cuda_common.h, src/runtime/cuda/,

### Solution 2 Pros/Cons/Workloads:

* **Pros**: Not depend on

* **Cons**: After unlinking with, the CI checks still cannot pass. [GPU
(When `-DUSE_TVM_OP=ON`) outputs the error message:

[2020-07-13T09:30:36.870Z] /usr/bin/ld: warning:, needed by /work/build/3rdparty/tvm/,
not found (try using -rpath or -rpath-link)
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuMemsetD32_v2'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuModuleLoadData'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuLaunchKernel'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuModuleGetFunction'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuModuleUnload'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuGetErrorName'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuDeviceGetName'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/ undefined reference
to `cuModuleGetGlobal_v2'
[2020-07-13T09:30:36.870Z] collect2: error: ld returned 1 exit status
[2020-07-13T09:30:36.870Z] ninja: build stopped: subcommand failed.
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,154 - root - INFO - Waiting for status of container
e95e5c4ca642 for 600 s.
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Container exit status:
{'Error': None, 'StatusCode': 1}
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - ERROR - Container exited with
an error 😞
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Executed command for reproduction:

It seems unlinking `` from `` is not enough and we should also unlink ``
from ``. However, `libtvm_runtime` does require `` during runtime.
if we remove the linkage on build instances, `tvm_runtime` is not able to run on test instances.
In order to fully address the problem, we have two options,

1. Build two versions of tvm, one links `` for compiling the tvm operators; another
does not link `` which will be transferred to test instances for tvmop tests.
2. We do `dlopen(“”)` in tvm.

* **Workloads**: 
    * option 1: ~1 week to modify tvm, 1.5-2 weeks modify CI
    * option 2: ~2 weeks to modify tvm
    * Both options require big surgery on TVM, thus contributing back to upstream tvm might
be difficult. Moreover, there’s also risk that tvm community would push back our suggestion.
Even though they agree to do so, it might cost ~2 weeks to upstream the changes, and another
~1.5 weeks to sync mxnet’s tvm with updated apache/tvm.

## Proposal:

Given the fact that it might take another 4 - 6 weeks to fully address the CI problem, we
propose to,

* Submit the fix for problem 1.
* Link stub/ and enable 1 instance for testing. If this is not acceptable, can we
keep disabling the tvmop CI for now as it is not an essential component?
* Open an issue/RFC in MXNet and TVM to track the remaining problems.

* * *

## Comments:

Also, When setting `-DUSE_TVM_OP=OFF` [the CI checks would be stuck](
The output of GPU CUDA RTC looks like:

[2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/tvm/
[2020-07-13T18:04:12.876Z] md5sum: build/3rdparty/tvm/ No such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/tvm/
[2020-07-13T18:04:12.876Z] ls: cannot access 'build/3rdparty/tvm/': No such
file or directory
[2020-07-13T18:04:12.876Z] + md5sum build/
[2020-07-13T18:04:12.876Z] md5sum: build/ No such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/
[2020-07-13T18:04:12.876Z] ls: cannot access 'build/': No such file or directory
[2020-07-13T18:04:12.876Z] + md5sum build/tvmop.conf
[2020-07-13T18:04:12.876Z] md5sum: build/tvmop.conf: No such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/tvmop.conf
[2020-07-13T18:04:12.876Z] ls: cannot access 'build/tvmop.conf': No such file or directory
[2020-07-13T18:04:12.876Z] + md5sum build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] 66aa8c8a37ffaaa9692ae98bda88491c  build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] + ls -lh build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 34M Jul 13 18:04 build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/openmp/runtime/src/
[2020-07-13T18:04:12.876Z] 819a0c986ae9e233b0a9525e71c906d9  build/3rdparty/openmp/runtime/src/
[2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/openmp/runtime/src/
[2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 1.1M Jul 13 17:56 build/3rdparty/openmp/runtime/src/
[2020-07-13T18:04:12.876Z] + return 0

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message