From dev-return-7774-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Wed Jul 15 06:51:45 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 606521804BB for ; Wed, 15 Jul 2020 08:51:45 +0200 (CEST) Received: (qmail 21329 invoked by uid 500); 15 Jul 2020 06:51:44 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 21312 invoked by uid 99); 15 Jul 2020 06:51:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2020 06:51:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 00BA318144F for ; Wed, 15 Jul 2020 06:51:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.9 X-Spam-Level: X-Spam-Status: No, score=-0.9 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.2, MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=github.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id lQOapv0zu-Pp for ; Wed, 15 Jul 2020 06:51:40 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::233; helo=mail-lj1-x233.google.com; envelope-from=dmlc.notification+caf_=dev=mxnet.apache.org@gmail.com; receiver= Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 7972A7F69A for ; Wed, 15 Jul 2020 06:51:39 +0000 (UTC) Received: by mail-lj1-x233.google.com with SMTP id e4so1355241ljn.4 for ; Tue, 14 Jul 2020 23:51:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:delivered-to:dkim-signature:date:from:reply-to :to:cc:message-id:subject:mime-version:content-transfer-encoding :precedence:list-id:list-archive:list-post:list-unsubscribe; bh=52W1P4T3dv2NnfVNFsvoNBV4+EMG9kqX2bNaWVa+hV0=; b=nu9vCdCNMB1nMRVpkNKYonVAuoyXoFiE8evj+vt9CI34C3Zd7S56AJZJ44yj9qMIUf ks1R7LLYJst24ysvDcKGdvfpHzq+2FSd6kgySAkFgfm8j7HSNlA8sRRWgTziT7uktAFO CXICZEziHJYe2fKM1uBSDCFC8to/gGlpX647nTpT8aFPynOMIEx0rSrTdx9T8yR5L6Zh +8LHnNxOWiHD0EQqQ9ATMRpdfzvq3BgAdO0MmyWrlaDKM+h7UzzrNeN7MDgW0yAvubT8 3daH4ir3G46QNG88cU08BeTuKHinedbOrFw8e1+mf89ykV77sqMcYi0XUbW9a5pmWy1G Cvdw== X-Gm-Message-State: AOAM532rOXtudUZEvE0la3hXpplnbfUmUubcWLePBGWoyw0Xl2RNltSR 6W7YR0RlIvlHDRI2/V5G/WHsMT2VWDlwl2cEMxwrCObQo9V7DK4= X-Received: by 2002:a05:651c:30c:: with SMTP id a12mr2603581ljp.410.1594795898577; Tue, 14 Jul 2020 23:51:38 -0700 (PDT) X-Forwarded-To: dev@mxnet.apache.org X-Forwarded-For: dmlc.notification@gmail.com dev@mxnet.apache.org Delivered-To: dmlc.notification@gmail.com Received: by 2002:ab3:5012:0:0:0:0:0 with SMTP id y18csp353645ltb; Tue, 14 Jul 2020 23:51:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzu/7bmiJr0uLCtF4VTe0DZmVJSUJE5N/pGCoz35xfk36fmpGxzC1/5/Trx7WQR+bFqIy34 X-Received: by 2002:a9d:d13:: with SMTP id 19mr7916242oti.83.1594795896414; Tue, 14 Jul 2020 23:51:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594795896; cv=none; d=google.com; s=arc-20160816; b=ec8kNRYUlgUQg4DZxSOQgakdP0ZgOBmhQw12qp+rvpCtjmyesin1iflEgZ7sklAQkN NuG3DJuThX3jts0zgtb/2lwXQNyzrHpWRUuOuOIws+1p1pQWVfhQKK0QIy0AhKPld6kP E5RZ9UW0QKfaBObmo2/8pCDRa5KkafdtQXLBiABLK0REs7EQ01lTwa5xZl/lZA4TcIHF KvKMqatyfr3W20eGWpBgMlbQOfne5PJlf8Wkn3XuX2XmwZ8xpzzysBSL4lX0KRO34j3p 90rh5dr2SjzV+yH28mTmWu3UrZ99AXOX3mknEJBLFPruEThQYLqb04KQI+dfNPrFod7X 1qbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-post:list-archive:list-id:precedence :content-transfer-encoding:mime-version:subject:message-id:cc:to :reply-to:from:date:dkim-signature; bh=52W1P4T3dv2NnfVNFsvoNBV4+EMG9kqX2bNaWVa+hV0=; b=f4kZmzJRtQ0jyVyg4h2hoZq1JbOgz/Vd/FUpl3NSs45ZMUtW79iABqxEJ2OjCqRpYK N92YuS+H/JcSipvrDCYyE45nDt9CEhAi5ORpON7n+xd5hNnHFewkyCjYuqhFpFUHbi95 n+g+C8QjqFrOI6CyH72D6JAy5TRYJpdeYwqb0nQ18bVxsWH5RmP293lR8tAGXvFwjuS3 tmXAPWdO5QsF4sjLYK/vzE4Axo2MEecYzarVodiO1np+KtdiJltkROfG1SRzvfRmUpyR iLZuiAl6+S7ospBj+CQ0ECSvzKH0K5xb8nSqIPr+QHx53q82zzuDreL4FpXU4P2doaQQ ms6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=HCFqJJEx; spf=pass (google.com: domain of noreply@github.com designates 192.30.252.203 as permitted sender) smtp.mailfrom=noreply@github.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com Received: from out-20.smtp.github.com (out-20.smtp.github.com. [192.30.252.203]) by mx.google.com with ESMTPS id p5si756617oig.230.2020.07.14.23.51.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Jul 2020 23:51:36 -0700 (PDT) Received-SPF: pass (google.com: domain of noreply@github.com designates 192.30.252.203 as permitted sender) client-ip=192.30.252.203; Authentication-Results: mx.google.com; dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=HCFqJJEx; spf=pass (google.com: domain of noreply@github.com designates 192.30.252.203 as permitted sender) smtp.mailfrom=noreply@github.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com Received: from github-lowworker-fa7043e.ash1-iad.github.net (github-lowworker-fa7043e.ash1-iad.github.net [10.56.109.45]) by smtp.github.com (Postfix) with ESMTP id D188D8C1E44 for ; Tue, 14 Jul 2020 23:51:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com; s=pf2014; t=1594795895; bh=52W1P4T3dv2NnfVNFsvoNBV4+EMG9kqX2bNaWVa+hV0=; h=Date:From:Reply-To:To:Cc:Subject:List-ID:List-Archive:List-Post: List-Unsubscribe:From; b=HCFqJJExZTQeRqjmRB2Q5NObhKyO4vgYtbUZ27bYf8vmbw6R5lrqkGYjw3rVjsZCB zVHYKnLAT86ahg4obPbWWSaQrWdhkt5WW87fgIMDfn2c02dr2blCCP5J/J9VcX2bRp apC6XQekq4thjRvC3qsD2Rv2X7nyG4w5pu3NWG0A= Date: Tue, 14 Jul 2020 23:51:35 -0700 From: Jinbo Ci Reply-To: apache/incubator-mxnet To: apache/incubator-mxnet Cc: Subscribed Message-ID: Subject: [apache/incubator-mxnet] [RFC] Use TVMOp with GPU & Build without libcuda.so in CI (#18716) Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="--==_mimepart_5f0ea777bf9aa_308a3faabeccd96035714"; charset=UTF-8 Content-Transfer-Encoding: 7bit X-GitHub-Sender: jinboci X-GitHub-Recipient: szha X-GitHub-Reason: subscribed List-Archive: https://github.com/apache/incubator-mxnet X-Auto-Response-Suppress: All X-GitHub-Recipient-Address: dmlc.notification@gmail.com ----==_mimepart_5f0ea777bf9aa_308a3faabeccd96035714 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable ## Problem 1: TVMOp doesn't work well with GPU builds [#17840](https://gi= thub.com/apache/incubator-mxnet/issues/17840)=0D =0D ### The error message:=0D =0D ```python=0D >>> import mxnet as mx=0D >>> x =3D mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=3Dmx.gpu())=0D >>> idx =3D x < 2=0D >>> x[idx]=0D Traceback (most recent call last):=0D File "", line 1, in =0D File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", l= ine 1013, in __lt__=0D return less(self, other)=0D File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", l= ine 8672, in less=0D return _mx_nd_np.less(x1, x2, out)=0D File "/home/ubuntu/Documents/mxnet/python/mxnet/ndarray/numpy/_op.py", = line 6869, in less=0D return _api_internal.less(x1, x2, out)=0D File "/home/ubuntu/Documents/mxnet/python/mxnet/_ffi/_ctypes/function.p= y", line 115, in __call__=0D raise get_last_ffi_error()=0D mxnet.base.MXNetError: Traceback (most recent call last):=0D File "../3rdparty/tvm/src/runtime/module.cc", line 125=0D File "../3rdparty/tvm/src/runtime/library_module.cc", line 94=0D TVMError: Check failed: ret =3D=3D 0 (-1 vs. 0) : Check failed: f !=3D nu= llptr: Cannot find function less_scalar_gpufloat32_2bool_2_kernel0 in the= imported modules or global registry=0D ```=0D =0D ### Root cause:=0D =0D In [mxnet/contrib/tvmop/compile.py](https://github.com/apache/incubator-m= xnet/blob/master/contrib/tvmop/compile.py), only `function_binary` `(llvm= Module)`is saved in `libtvmop.so`. The `imported_modules` `(cuda Module)= `is not saved. So TVM cannot import any gpu functions and cannot find `le= ss_scalar_gpufloat32_2bool_2_kernel0.`=0D =0D ```python=0D (Pdb) func_binary=0D Module(llvm, 55d7ce519d48)=0D (Pdb) func_binary.imported_modules[0]=0D Module(cuda, 55d7c7a09818)=0D ```=0D =0D ### Solution ([Github PR](https://github.com/apache/incubator-mxnet/pull/= 18678)):=0D =0D * Save `imported_modules[0]` to `libtvmop.cubin:`=0D * Define `Import` function (using `TVMOpModule->Import` ):=0D * Import `cubin_module` to `global_module`:=0D * Outputs:=0D =0D ```python=0D >>> import mxnet as mx=0D >>> x =3D mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=3Dmx.gpu())=0D [10:19:41] ../src/base.cc:80: cuDNN lib mismatch: linked-against version = 7605 !=3D compiled-against version 7501. Set MXNET_CUDNN_LIB_CHECKING=3D0= to quiet this warning.=0D [10:19:41] ../src/base.cc:84: Upgrade advisory: this mxnet has been built= against cuDNN lib version 7501, which is older than the oldest version t= ested by CI (7600). Set MXNET_CUDNN_LIB_CHECKING=3D0 to quiet this warnin= g.=0D >>> idx =3D x < 2=0D >>> x[idx]=0D array([0., 1., 1., 1.], ctx=3Dgpu(0))=0D ```=0D =0D ## Problem 2: CI Checks: libcuda.so does exist on the machine builds mxne= t=0D =0D ### The error message:=0D =0D When running [unix-gpu](http://jenkins.mxnet-ci.amazon-ml.com/blue/organi= zations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18526/18/pipeline) = checks:=0D =0D ```python=0D [2020-06-18T08:26:18.355Z] Traceback (most recent call last):=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/contrib/tvmop/compile.py",= line 20, in =0D [2020-06-18T08:26:18.355Z] import tvm=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/__= init__.py", line 27, in =0D [2020-06-18T08:26:18.355Z] from . import tensor=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/te= nsor.py", line 20, in =0D [2020-06-18T08:26:18.355Z] from ._ffi.object import Object, register_= object, ObjectGeneric, \=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_f= fi/object.py", line 24, in =0D [2020-06-18T08:26:18.355Z] from .base import _FFI_MODE, _RUNTIME_ONLY= , check_call, _LIB, c_str=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_f= fi/base.py", line 65, in =0D [2020-06-18T08:26:18.355Z] _LIB, _LIB_NAME =3D _load_lib()=0D [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_f= fi/base.py", line 57, in _load_lib=0D [2020-06-18T08:26:18.355Z] lib =3D ctypes.CDLL(lib_path[0], ctypes.RT= LD_GLOBAL)=0D [2020-06-18T08:26:18.355Z] File "/usr/lib/python3.6/ctypes/__init__.py"= , line 348, in __init__=0D [2020-06-18T08:26:18.355Z] self._handle =3D _dlopen(self._name, mode)= =0D [2020-06-18T08:26:18.355Z] OSError: libcuda.so.1: cannot open shared obje= ct file: No such file or directory=0D ```=0D =0D ### Root cause:=0D =0D The unix-gpu machine that **builds** mxnet does not have libcuda.so=0D =0D ### Solution 1:=0D =0D Link `libtvm.so` with `stub/libcuda.so` on the machine that **builds** CI= Checks.=0D =0D ### Solution 1 Pros/Cons/Workloads:=0D =0D * **Pros**: Solve the issue easily.=0D * **Cons**: Violates the effort of removing `libcuda.so` totally, (would = be great if someone can elaborate the motivation behind it).=0D * **Workloads**: ~1 week=0D =0D ### Solution 2 (Possible) ([Github PR](https://github.com/jinboci/incubat= or-tvm/pull/2)) :=0D =0D TVM links libcuda.so because it invokes CUDA driver API during runtime. W= hile these functions are not executed during compile-time. Therefore it i= s possible to remove them for compile-only purpose. =0D I have made a prototype to remove the linkage of libcuda.so from libtvm.s= o:=0D =0D * Set `target_link_libraries` of tvm and tvm_runtime differently. (CMakeL= ists.txt)=0D * Set an variable `CUDA_COMPILE_ONLY` to be `ON` to indicate =E2=80=9CBui= lding libtvm.so without libcuda.so=E2=80=9D (CMakeLists.txt)=0D * When `CUDA_COMPILE_ONLY` is `ON`, add compilation definition `-DCUDA_CO= MPILE_ONLY` (CMakeLists.txt)=0D * When `CUDA_COMPILE_ONLY` is defined (when compiling libtvm.so), ignore = any cuXXX CUDA Driver API functions: (cmake/modules/CUDA.cmake, src/runti= me/cuda/cuda_common.h, src/runtime/cuda/cuda_device_api.cc, src/runtime/c= uda/cuda_module.cc)=0D =0D ### Solution 2 Pros/Cons/Workloads:=0D =0D * **Pros**: Not depend on libcuda.so=0D =0D * **Cons**: After unlinking libtvm.so with libcuda.so, the CI checks stil= l cannot pass. [GPU CUDA RTC](http://jenkins.mxnet-ci.amazon-ml.com/blue/= organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18678/8/pipel= ine) (When `-DUSE_TVM_OP=3DON`) outputs the error message:=0D =0D ```python=0D [2020-07-13T09:30:36.870Z] /usr/bin/ld: warning: libcuda.so.1, needed by = /work/build/3rdparty/tvm/libtvm_runtime.so, not found (try using -rpath o= r -rpath-link)=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuMemsetD32_v2'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuModuleLoadData'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuLaunchKernel'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuModuleGetFunction'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuModuleUnload'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuGetErrorName'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuDeviceGetName'=0D [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: un= defined reference to `cuModuleGetGlobal_v2'=0D [2020-07-13T09:30:36.870Z] collect2: error: ld returned 1 exit status=0D [2020-07-13T09:30:36.870Z] ninja: build stopped: subcommand failed.=0D [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,154 - root - INFO - Waitin= g for status of container e95e5c4ca642 for 600 s.=0D [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Contai= ner exit status: {'Error': None, 'StatusCode': 1}=0D [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - ERROR - Conta= iner exited with an error =F0=9F=98=9E=0D [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Execut= ed command for reproduction:=0D [2020-07-13T09:30:36.870Z]=0D ```=0D =0D It seems unlinking `libcuda.so` from `libtvm.so` is not enough and we sho= uld also unlink `libcuda.so` from `libtvm_runtime.so`. However, `libtvm_r= untime` does require `libcuda.so` during runtime. if we remove the linkag= e on build instances, `tvm_runtime` is not able to run on test instances.= =0D In order to fully address the problem, we have two options,=0D =0D 1. Build two versions of tvm, one links `libcuda.so` for compiling the tv= m operators; another does not link `libcuda.so` which will be transferred= to test instances for tvmop tests.=0D 2. We do `dlopen(=E2=80=9Clibcuda.so=E2=80=9D)` in tvm.=0D =0D * **Workloads**: =0D * option 1: ~1 week to modify tvm, 1.5-2 weeks modify CI=0D * option 2: ~2 weeks to modify tvm=0D * Both options require big surgery on TVM, thus contributing back to = upstream tvm might be difficult. Moreover, there=E2=80=99s also risk that= tvm community would push back our suggestion. Even though they agree to = do so, it might cost ~2 weeks to upstream the changes, and another ~1.5 w= eeks to sync mxnet=E2=80=99s tvm with updated apache/tvm.=0D =0D ## Proposal:=0D =0D Given the fact that it might take another 4 - 6 weeks to fully address th= e CI problem, we propose to,=0D =0D * Submit the fix for problem 1.=0D * Link stub/libcuda.so and enable 1 instance for testing. If this is not = acceptable, can we keep disabling the tvmop CI for now as it is not an es= sential component?=0D * Open an issue/RFC in MXNet and TVM to track the remaining problems.=0D =0D * * *=0D =0D ## Comments:=0D =0D Also, When setting `-DUSE_TVM_OP=3DOFF` [the CI checks would be stuck](ht= tp://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-vali= dation%2Funix-gpu/detail/PR-18678/10/pipeline). The output of GPU CUDA RT= C looks like:=0D =0D ```python=0D [2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/tvm/libtvm_runtime.so=0D= [2020-07-13T18:04:12.876Z] md5sum: build/3rdparty/tvm/libtvm_runtime.so: = No such file or directory=0D [2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/tvm/libtvm_runtime.so=0D= [2020-07-13T18:04:12.876Z] ls: cannot access 'build/3rdparty/tvm/libtvm_r= untime.so': No such file or directory=0D [2020-07-13T18:04:12.876Z] + md5sum build/libtvmop.so=0D [2020-07-13T18:04:12.876Z] md5sum: build/libtvmop.so: No such file or dir= ectory=0D [2020-07-13T18:04:12.876Z] + ls -lh build/libtvmop.so=0D [2020-07-13T18:04:12.876Z] ls: cannot access 'build/libtvmop.so': No such= file or directory=0D [2020-07-13T18:04:12.876Z] + md5sum build/tvmop.conf=0D [2020-07-13T18:04:12.876Z] md5sum: build/tvmop.conf: No such file or dire= ctory=0D [2020-07-13T18:04:12.876Z] + ls -lh build/tvmop.conf=0D [2020-07-13T18:04:12.876Z] ls: cannot access 'build/tvmop.conf': No such = file or directory=0D [2020-07-13T18:04:12.876Z] + md5sum build/tests/mxnet_unit_tests=0D [2020-07-13T18:04:12.876Z] 66aa8c8a37ffaaa9692ae98bda88491c build/tests/= mxnet_unit_tests=0D [2020-07-13T18:04:12.876Z] + ls -lh build/tests/mxnet_unit_tests=0D [2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 34M J= ul 13 18:04 build/tests/mxnet_unit_tests=0D [2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/openmp/runtime/src/lib= omp.so=0D [2020-07-13T18:04:12.876Z] 819a0c986ae9e233b0a9525e71c906d9 build/3rdpar= ty/openmp/runtime/src/libomp.so=0D [2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/openmp/runtime/src/lib= omp.so=0D [2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 1.1M = Jul 13 17:56 build/3rdparty/openmp/runtime/src/libomp.so=0D [2020-07-13T18:04:12.876Z] + return 0=0D ```=0D =0D =0D -- =0D You are receiving this because you are subscribed to this thread.=0D Reply to this email directly or view it on GitHub:=0D https://github.com/apache/incubator-mxnet/issues/18716= ----==_mimepart_5f0ea777bf9aa_308a3faabeccd96035714--