mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diego Rodriguez-Losada <diego.rlos...@gmail.com>
Subject Re: using conan to manage Apache Incubator MXNet project dependencies
Date Tue, 27 Nov 2018 13:23:57 GMT
Hi Pedro,

Conan is distributed. So besides building from sources the dependencies, it
is also possible to create binaries yourself for those dependencies (with
the existing recipes, or your own recipes), and host them in your own repo
(Bintray OSS repo, or Artifactory).

This will provide both the security that you own the dependencies binaries
and the convenience and speed of not having to build from sources. Even if
you provide the binaries, consumers can always fallback to build from
sources too.

Kind regards,
Diego

El mar., 27 nov. 2018 a las 13:34, Konstantin Ivlev (<tomskside@gmail.com>)
escribió:

> Hi Pedro,
>
> yes, you're absolutely right, by default, conan will be pulling prebuilt
> binaries for the libraries from the bintray.
> however, if prebuilt binaries are not available (e.g. because you use some
> different compiler for which we don't have prebuilt binaries),
> or if you want to build binaries yourself for some another reason,
> then libraries always might be built from source (by passing e.g. "--build
> always", "--build missing" or "--build <library>" to the conan install
> command line).
>
> yours sincerely, Konstantin
>
> вт, 27 нояб. 2018 г. в 19:27, Pedro Larroy <pedro.larroy.lists@gmail.com>:
>
> > Hi Konstantin
> >
> > Thanks for this contribution. With your proposed changes, when
> > building MXNet we will be pulling binaries for the libraries managed
> > by conan?
> >
> >
> > Pedro.
> > On Mon, Nov 26, 2018 at 11:43 AM Konstantin Ivlev <tomskside@gmail.com>
> > wrote:
> > >
> > > Hello, Ivan,
> > >
> > > could you possibly clarify your question (may be explaining the
> use-case
> > > behind it)?
> > > Gradle appears to be build system, AFAIK more popular in Java world.
> > > meanwhile, Apache Incubator MXNet project uses CMake as its build
> system.
> > > please correct me, I am wrong.
> > > in general, conan, as a package manager, is pretty
> build-system-agnostic,
> > > and it may work with arbitrary build systems (to count a few, CMake,
> > > premake, qmake, full list:
> > > https://docs.conan.io/en/latest/reference/generators.html). I don't
> > think
> > > Gradle is exception here.
> > > also, for instance, Android Studio also uses Gradle for Android C++
> > > projects, and conan works flawlessly in this particular case (
> > >
> https://blog.conan.io/2018/02/13/Android-Studio-project-Conan-Boost.html
> > ).
> > >
> > > yours sincerely, Konstantin
> > >
> > > пн, 26 нояб. 2018 г. в 16:43, Ivan Serdyuk <
> local.tourist.kiev@gmail.com
> > >:
> > >
> > > > Kostantin, and what (overall) option with using Gradle? Does this
> your
> > > > suggested package manager has been supported by Gradle?
> > > >
> > > > Ivan
> > > >
> > > > On Mon, Nov 26, 2018 at 9:43 AM Konstantin Ivlev <
> tomskside@gmail.com>
> > > > wrote:
> > > >
> > > > > hello,
> > > > >
> > > > > this email is related to the following PR and JIRA ticket:
> > > > > - [MXNET-1229] use OpenBLAS, lapack & OpenCV from conan
> > > > > <https://github.com/apache/incubator-mxnet/pull/13400>
> > > > > - use conan to manage project dependencies
> > > > > <https://issues.apache.org/jira/browse/MXNET-1229>
> > > > >
> > > > > conan <https://conan.io> is an open-source package manager
for C++
> > > > > projects. it allows to manage project dependencies in transparent
> and
> > > > > declarative manner.
> > > > >
> > > > > currently, apache incubator-mxnet project uses the following
> > different
> > > > > ways to manage its dependencies:
> > > > >
> > > > > - download GitHub archives during the build
> > > > >     - OpenBLAS <https://github.com/xianyi/OpenBLAS>
> > > > >     - OpenCV <https://github.com/opencv/opencv>
> > > > > - conda <https://conda.io/> (alternative way to GitHub archives)
> > > > > - download from CMake
> > > > >     - Intel Math Kernel Library <
> > https://software.intel.com/en-us/mkl>
> > > > > (MKL)
> > > > > - Git submodules
> > > > >     - cub <https://github.com/dmlc/cub>
> > > > >     - dlpack <https://github.com/dmlc/dlpack>
> > > > >     - dmlc-core <https://github.com/dmlc/dmlc-core>
> > > > >     - googletest <https://github.com/abseil/googletest>
> > > > >     - mkldnn <https://github.com/intel/mkl-dnn>
> > > > >     - mshadow <https://github.com/dmlc/mshadow>
> > > > >     - onnx-tensorrt <https://github.com/onnx/onnx-tensorrt>
> > > > >     - openmp <https://github.com/llvm-mirror/openmp>
> > > > >     - ps-lite <https://github.com/dmlc/ps-lite>
> > > > >     - tvm <https://github.com/dmlc/tvm>
> > > > >
> > > > > this appears to be very heterogeneous and hard to manage/maintain,
> as
> > > > > multiple various commands are in use to achieve dependencies
> > > > installation,
> > > > > as well as multiple places are to look for dependency versions and
> > their
> > > > > updates.
> > > > >
> > > > > with conan, it may became much more straightforward, as
> dependencies
> > will
> > > > > be declared in single place (conanfile) and installed via single
> > command
> > > > > (conan install).
> > > > >
> > > > > as project is very complex, and has lots of dependencies, for the
> > first
> > > > > prototype I've used only very few of dependencies from conan:
> OpenCV
> > > > > <https://bintray.com/conan-community/conan/opencv%3Aconan>,
> OpenBLAS
> > > > > <https://bintray.com/conan-community/conan/openblas%3Aconan>
and
> > lapack
> > > > > <https://bintray.com/conan-community/conan/lapack%3Aconan>.
> > > > > others may be easily added then one by one, but they first has to
> be
> > > > > packaged (not all of them are packaged yet, e.g. GoogleTest
> > > > > <https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters>
> > is
> > > > > available, while MKL is not).
> > > > >
> > > > > I attach patch which adds an initial conan support as proof of
> > concept.
> > > > > also, I attach two simple build scripts, which I've used to test
> (for
> > > > > Windows and Linux / Mac OS X). Google Mail blocks .sh and .cmd
> > > > extensions,
> > > > > so you'll need to rename files.
> > > > > lemme know if you have any further questions.
> > > > >
> > > > > yours sincerely, Konstantin
> > > > >
> > > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message