mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Larroy <pedro.larroy.li...@gmail.com>
Subject Re: using conan to manage Apache Incubator MXNet project dependencies
Date Tue, 27 Nov 2018 14:17:24 GMT
Thanks both for the detailed explanations. Couple of more questions:

Is it easy to create a build which will build dependencies from source?
What guarantees you get with conan with regards to ABI / C++ stdlib
binary compatibility of the pulled dependencies?

Just to clarify: My concerns are in terms of reproducible builds /
source only distribution and undefined behaviour due to different
compiler / stdlib versions. Are these valid or is it oudated
knowledge?

Pedro.
On Tue, Nov 27, 2018 at 2:34 PM Diego Rodriguez-Losada
<diego.rlosada@gmail.com> wrote:
>
> Hi Pedro,
>
> Conan is distributed. So besides building from sources the dependencies, it
> is also possible to create binaries yourself for those dependencies (with
> the existing recipes, or your own recipes), and host them in your own repo
> (Bintray OSS repo, or Artifactory).
>
> This will provide both the security that you own the dependencies binaries
> and the convenience and speed of not having to build from sources. Even if
> you provide the binaries, consumers can always fallback to build from
> sources too.
>
> Kind regards,
> Diego
>
> El mar., 27 nov. 2018 a las 13:34, Konstantin Ivlev (<tomskside@gmail.com>)
> escribió:
>
> > Hi Pedro,
> >
> > yes, you're absolutely right, by default, conan will be pulling prebuilt
> > binaries for the libraries from the bintray.
> > however, if prebuilt binaries are not available (e.g. because you use some
> > different compiler for which we don't have prebuilt binaries),
> > or if you want to build binaries yourself for some another reason,
> > then libraries always might be built from source (by passing e.g. "--build
> > always", "--build missing" or "--build <library>" to the conan install
> > command line).
> >
> > yours sincerely, Konstantin
> >
> > вт, 27 нояб. 2018 г. в 19:27, Pedro Larroy <pedro.larroy.lists@gmail.com>:
> >
> > > Hi Konstantin
> > >
> > > Thanks for this contribution. With your proposed changes, when
> > > building MXNet we will be pulling binaries for the libraries managed
> > > by conan?
> > >
> > >
> > > Pedro.
> > > On Mon, Nov 26, 2018 at 11:43 AM Konstantin Ivlev <tomskside@gmail.com>
> > > wrote:
> > > >
> > > > Hello, Ivan,
> > > >
> > > > could you possibly clarify your question (may be explaining the
> > use-case
> > > > behind it)?
> > > > Gradle appears to be build system, AFAIK more popular in Java world.
> > > > meanwhile, Apache Incubator MXNet project uses CMake as its build
> > system.
> > > > please correct me, I am wrong.
> > > > in general, conan, as a package manager, is pretty
> > build-system-agnostic,
> > > > and it may work with arbitrary build systems (to count a few, CMake,
> > > > premake, qmake, full list:
> > > > https://docs.conan.io/en/latest/reference/generators.html). I don't
> > > think
> > > > Gradle is exception here.
> > > > also, for instance, Android Studio also uses Gradle for Android C++
> > > > projects, and conan works flawlessly in this particular case (
> > > >
> > https://blog.conan.io/2018/02/13/Android-Studio-project-Conan-Boost.html
> > > ).
> > > >
> > > > yours sincerely, Konstantin
> > > >
> > > > пн, 26 нояб. 2018 г. в 16:43, Ivan Serdyuk <
> > local.tourist.kiev@gmail.com
> > > >:
> > > >
> > > > > Kostantin, and what (overall) option with using Gradle? Does this
> > your
> > > > > suggested package manager has been supported by Gradle?
> > > > >
> > > > > Ivan
> > > > >
> > > > > On Mon, Nov 26, 2018 at 9:43 AM Konstantin Ivlev <
> > tomskside@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > hello,
> > > > > >
> > > > > > this email is related to the following PR and JIRA ticket:
> > > > > > - [MXNET-1229] use OpenBLAS, lapack & OpenCV from conan
> > > > > > <https://github.com/apache/incubator-mxnet/pull/13400>
> > > > > > - use conan to manage project dependencies
> > > > > > <https://issues.apache.org/jira/browse/MXNET-1229>
> > > > > >
> > > > > > conan <https://conan.io> is an open-source package manager
for C++
> > > > > > projects. it allows to manage project dependencies in transparent
> > and
> > > > > > declarative manner.
> > > > > >
> > > > > > currently, apache incubator-mxnet project uses the following
> > > different
> > > > > > ways to manage its dependencies:
> > > > > >
> > > > > > - download GitHub archives during the build
> > > > > >     - OpenBLAS <https://github.com/xianyi/OpenBLAS>
> > > > > >     - OpenCV <https://github.com/opencv/opencv>
> > > > > > - conda <https://conda.io/> (alternative way to GitHub
archives)
> > > > > > - download from CMake
> > > > > >     - Intel Math Kernel Library <
> > > https://software.intel.com/en-us/mkl>
> > > > > > (MKL)
> > > > > > - Git submodules
> > > > > >     - cub <https://github.com/dmlc/cub>
> > > > > >     - dlpack <https://github.com/dmlc/dlpack>
> > > > > >     - dmlc-core <https://github.com/dmlc/dmlc-core>
> > > > > >     - googletest <https://github.com/abseil/googletest>
> > > > > >     - mkldnn <https://github.com/intel/mkl-dnn>
> > > > > >     - mshadow <https://github.com/dmlc/mshadow>
> > > > > >     - onnx-tensorrt <https://github.com/onnx/onnx-tensorrt>
> > > > > >     - openmp <https://github.com/llvm-mirror/openmp>
> > > > > >     - ps-lite <https://github.com/dmlc/ps-lite>
> > > > > >     - tvm <https://github.com/dmlc/tvm>
> > > > > >
> > > > > > this appears to be very heterogeneous and hard to manage/maintain,
> > as
> > > > > > multiple various commands are in use to achieve dependencies
> > > > > installation,
> > > > > > as well as multiple places are to look for dependency versions
and
> > > their
> > > > > > updates.
> > > > > >
> > > > > > with conan, it may became much more straightforward, as
> > dependencies
> > > will
> > > > > > be declared in single place (conanfile) and installed via single
> > > command
> > > > > > (conan install).
> > > > > >
> > > > > > as project is very complex, and has lots of dependencies, for
the
> > > first
> > > > > > prototype I've used only very few of dependencies from conan:
> > OpenCV
> > > > > > <https://bintray.com/conan-community/conan/opencv%3Aconan>,
> > OpenBLAS
> > > > > > <https://bintray.com/conan-community/conan/openblas%3Aconan>
and
> > > lapack
> > > > > > <https://bintray.com/conan-community/conan/lapack%3Aconan>.
> > > > > > others may be easily added then one by one, but they first has
to
> > be
> > > > > > packaged (not all of them are packaged yet, e.g. GoogleTest
> > > > > > <https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters>
> > > is
> > > > > > available, while MKL is not).
> > > > > >
> > > > > > I attach patch which adds an initial conan support as proof
of
> > > concept.
> > > > > > also, I attach two simple build scripts, which I've used to
test
> > (for
> > > > > > Windows and Linux / Mac OS X). Google Mail blocks .sh and .cmd
> > > > > extensions,
> > > > > > so you'll need to rename files.
> > > > > > lemme know if you have any further questions.
> > > > > >
> > > > > > yours sincerely, Konstantin
> > > > > >
> > > > >
> > >
> >

Mime
View raw message