mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Ivlev <tomsks...@gmail.com>
Subject using conan to manage Apache Incubator MXNet project dependencies
Date Mon, 26 Nov 2018 07:42:59 GMT
hello,

this email is related to the following PR and JIRA ticket:
- [MXNET-1229] use OpenBLAS, lapack & OpenCV from conan
<https://github.com/apache/incubator-mxnet/pull/13400>
- use conan to manage project dependencies
<https://issues.apache.org/jira/browse/MXNET-1229>

conan <https://conan.io> is an open-source package manager for C++
projects. it allows to manage project dependencies in transparent and
declarative manner.

currently, apache incubator-mxnet project uses the following different ways
to manage its dependencies:

- download GitHub archives during the build
    - OpenBLAS <https://github.com/xianyi/OpenBLAS>
    - OpenCV <https://github.com/opencv/opencv>
- conda <https://conda.io/> (alternative way to GitHub archives)
- download from CMake
    - Intel Math Kernel Library <https://software.intel.com/en-us/mkl> (MKL)
- Git submodules
    - cub <https://github.com/dmlc/cub>
    - dlpack <https://github.com/dmlc/dlpack>
    - dmlc-core <https://github.com/dmlc/dmlc-core>
    - googletest <https://github.com/abseil/googletest>
    - mkldnn <https://github.com/intel/mkl-dnn>
    - mshadow <https://github.com/dmlc/mshadow>
    - onnx-tensorrt <https://github.com/onnx/onnx-tensorrt>
    - openmp <https://github.com/llvm-mirror/openmp>
    - ps-lite <https://github.com/dmlc/ps-lite>
    - tvm <https://github.com/dmlc/tvm>

this appears to be very heterogeneous and hard to manage/maintain, as
multiple various commands are in use to achieve dependencies installation,
as well as multiple places are to look for dependency versions and their
updates.

with conan, it may became much more straightforward, as dependencies will
be declared in single place (conanfile) and installed via single command
(conan install).

as project is very complex, and has lots of dependencies, for the first
prototype I've used only very few of dependencies from conan: OpenCV
<https://bintray.com/conan-community/conan/opencv%3Aconan>, OpenBLAS
<https://bintray.com/conan-community/conan/openblas%3Aconan> and lapack
<https://bintray.com/conan-community/conan/lapack%3Aconan>.
others may be easily added then one by one, but they first has to be
packaged (not all of them are packaged yet, e.g. GoogleTest
<https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters> is
available, while MKL is not).

I attach patch which adds an initial conan support as proof of concept.
also, I attach two simple build scripts, which I've used to test (for
Windows and Linux / Mac OS X). Google Mail blocks .sh and .cmd extensions,
so you'll need to rename files.
lemme know if you have any further questions.

yours sincerely, Konstantin

Mime
View raw message