mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Ivlev <tomsks...@gmail.com>
Subject Re: using conan to manage Apache Incubator MXNet project dependencies
Date Wed, 28 Nov 2018 15:27:12 GMT
> Is it possible to have an offline build somehow? For example, if all
> dependencies would be stored locally. Probably, that would require some
> modifications to the conan file, right?
as soon as all dependencies are stored locally,
conan will use binaries from the local cache.
no modifications are required to conan file or workflow in this case.

ср, 28 нояб. 2018 г. в 22:24, Anton Chernov <mechernov@gmail.com>:

> Great, thank you for your answers.
>
> Is it possible to have an offline build somehow? For example, if all
> dependencies would be stored locally. Probably, that would require some
> modifications to the conan file, right?
>
>
> вт, 27 нояб. 2018 г. в 16:59, Konstantin Ivlev <tomskside@gmail.com>:
>
> > >  Would it be possible to define cmake itself as a conan dependency for
> > MXNet?
> > yes, it is definitely possible
> > for instance, to declare CMake 3.13.0 as a dependency, the following line
> > has to be added to the conanfile.py:
> >
> > build_requires = "cmake_installer/3.13.0@conan/stable"
> >
> > other build tools might be added in a similar manner.
> >
> > вт, 27 нояб. 2018 г. в 22:54, Anton Chernov <mechernov@gmail.com>:
> >
> > > I think I asked this already, but want to confirm in regards to the
> > > following discussion:
> > >
> > > MXNet CMake build - raise minimal required version
> > >
> > >
> >
> https://lists.apache.org/thread.html/09772f9447ebff72cdfea5512ae70e295db8a21ba0ddb39359cd0a77@%3Cdev.mxnet.apache.org%3E
> > >
> > > Would it be possible to define cmake itself as a conan dependency for
> > > MXNet?
> > >
> > > Best
> > > Anton
> > >
> > > вт, 27 нояб. 2018 г. в 15:44, Konstantin Ivlev <tomskside@gmail.com>:
> > >
> > > > both questions are totally valid.
> > > > > Is it easy to create a build which will build dependencies from
> > source?
> > > > 1. yep, it's very easy, just add `--build` argument to the `conan
> > > install`
> > > > command line
> > > > >  What guarantees you get with conan with regards to ABI / C++
> stdlib
> > > > binary compatibility of the pulled dependencies?
> > > > 2. the default binary compatibility model may be found in
> settings.yml
> > (
> > > >
> > > >
> > >
> >
> https://github.com/conan-io/conan/blob/develop/conans/client/conf/__init__.py#L14
> > > > )
> > > > it includes the operation system, architecture, compiler, its version
> > and
> > > > C++ stdlib.
> > > > while it's true that it won't cover 100% of use-cases (e.g. for older
> > > > distros or older glibc), it's very good default for most users.
> > > > you may always define custom binary compatibility attributes in your
> > > > settings.yml (e.g. distro, microarchitecture, sanitizers, etc.).
> > > >
> > > >
> > > > вт, 27 нояб. 2018 г. в 21:17, Pedro Larroy <
> > pedro.larroy.lists@gmail.com
> > > >:
> > > >
> > > > > Thanks both for the detailed explanations. Couple of more
> questions:
> > > > >
> > > > > Is it easy to create a build which will build dependencies from
> > source?
> > > > > What guarantees you get with conan with regards to ABI / C++ stdlib
> > > > > binary compatibility of the pulled dependencies?
> > > > >
> > > > > Just to clarify: My concerns are in terms of reproducible builds
/
> > > > > source only distribution and undefined behaviour due to different
> > > > > compiler / stdlib versions. Are these valid or is it oudated
> > > > > knowledge?
> > > > >
> > > > > Pedro.
> > > > > On Tue, Nov 27, 2018 at 2:34 PM Diego Rodriguez-Losada
> > > > > <diego.rlosada@gmail.com> wrote:
> > > > > >
> > > > > > Hi Pedro,
> > > > > >
> > > > > > Conan is distributed. So besides building from sources the
> > > > dependencies,
> > > > > it
> > > > > > is also possible to create binaries yourself for those
> dependencies
> > > > (with
> > > > > > the existing recipes, or your own recipes), and host them in
your
> > own
> > > > > repo
> > > > > > (Bintray OSS repo, or Artifactory).
> > > > > >
> > > > > > This will provide both the security that you own the dependencies
> > > > > binaries
> > > > > > and the convenience and speed of not having to build from
> sources.
> > > Even
> > > > > if
> > > > > > you provide the binaries, consumers can always fallback to build
> > from
> > > > > > sources too.
> > > > > >
> > > > > > Kind regards,
> > > > > > Diego
> > > > > >
> > > > > > El mar., 27 nov. 2018 a las 13:34, Konstantin Ivlev (<
> > > > > tomskside@gmail.com>)
> > > > > > escribió:
> > > > > >
> > > > > > > Hi Pedro,
> > > > > > >
> > > > > > > yes, you're absolutely right, by default, conan will be
pulling
> > > > > prebuilt
> > > > > > > binaries for the libraries from the bintray.
> > > > > > > however, if prebuilt binaries are not available (e.g. because
> you
> > > use
> > > > > some
> > > > > > > different compiler for which we don't have prebuilt binaries),
> > > > > > > or if you want to build binaries yourself for some another
> > reason,
> > > > > > > then libraries always might be built from source (by passing
> e.g.
> > > > > "--build
> > > > > > > always", "--build missing" or "--build <library>"
to the conan
> > > > install
> > > > > > > command line).
> > > > > > >
> > > > > > > yours sincerely, Konstantin
> > > > > > >
> > > > > > > вт, 27 нояб. 2018 г. в 19:27, Pedro Larroy <
> > > > > pedro.larroy.lists@gmail.com>:
> > > > > > >
> > > > > > > > Hi Konstantin
> > > > > > > >
> > > > > > > > Thanks for this contribution. With your proposed changes,
> when
> > > > > > > > building MXNet we will be pulling binaries for the
libraries
> > > > managed
> > > > > > > > by conan?
> > > > > > > >
> > > > > > > >
> > > > > > > > Pedro.
> > > > > > > > On Mon, Nov 26, 2018 at 11:43 AM Konstantin Ivlev
<
> > > > > tomskside@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hello, Ivan,
> > > > > > > > >
> > > > > > > > > could you possibly clarify your question (may
be explaining
> > the
> > > > > > > use-case
> > > > > > > > > behind it)?
> > > > > > > > > Gradle appears to be build system, AFAIK more
popular in
> Java
> > > > > world.
> > > > > > > > > meanwhile, Apache Incubator MXNet project uses
CMake as its
> > > build
> > > > > > > system.
> > > > > > > > > please correct me, I am wrong.
> > > > > > > > > in general, conan, as a package manager, is pretty
> > > > > > > build-system-agnostic,
> > > > > > > > > and it may work with arbitrary build systems
(to count a
> few,
> > > > > CMake,
> > > > > > > > > premake, qmake, full list:
> > > > > > > > > https://docs.conan.io/en/latest/reference/generators.html
> ).
> > I
> > > > > don't
> > > > > > > > think
> > > > > > > > > Gradle is exception here.
> > > > > > > > > also, for instance, Android Studio also uses
Gradle for
> > Android
> > > > C++
> > > > > > > > > projects, and conan works flawlessly in this
particular
> case
> > (
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://blog.conan.io/2018/02/13/Android-Studio-project-Conan-Boost.html
> > > > > > > > ).
> > > > > > > > >
> > > > > > > > > yours sincerely, Konstantin
> > > > > > > > >
> > > > > > > > > пн, 26 нояб. 2018 г. в 16:43, Ivan Serdyuk
<
> > > > > > > local.tourist.kiev@gmail.com
> > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Kostantin, and what (overall) option with
using Gradle?
> > Does
> > > > this
> > > > > > > your
> > > > > > > > > > suggested package manager has been supported
by Gradle?
> > > > > > > > > >
> > > > > > > > > > Ivan
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 26, 2018 at 9:43 AM Konstantin
Ivlev <
> > > > > > > tomskside@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > hello,
> > > > > > > > > > >
> > > > > > > > > > > this email is related to the following
PR and JIRA
> > ticket:
> > > > > > > > > > > - [MXNET-1229] use OpenBLAS, lapack
& OpenCV from conan
> > > > > > > > > > > <https://github.com/apache/incubator-mxnet/pull/13400>
> > > > > > > > > > > - use conan to manage project dependencies
> > > > > > > > > > > <https://issues.apache.org/jira/browse/MXNET-1229>
> > > > > > > > > > >
> > > > > > > > > > > conan <https://conan.io> is an
open-source package
> > manager
> > > > > for C++
> > > > > > > > > > > projects. it allows to manage project
dependencies in
> > > > > transparent
> > > > > > > and
> > > > > > > > > > > declarative manner.
> > > > > > > > > > >
> > > > > > > > > > > currently, apache incubator-mxnet project
uses the
> > > following
> > > > > > > > different
> > > > > > > > > > > ways to manage its dependencies:
> > > > > > > > > > >
> > > > > > > > > > > - download GitHub archives during the
build
> > > > > > > > > > >     - OpenBLAS <https://github.com/xianyi/OpenBLAS>
> > > > > > > > > > >     - OpenCV <https://github.com/opencv/opencv>
> > > > > > > > > > > - conda <https://conda.io/> (alternative
way to GitHub
> > > > > archives)
> > > > > > > > > > > - download from CMake
> > > > > > > > > > >     - Intel Math Kernel Library <
> > > > > > > > https://software.intel.com/en-us/mkl>
> > > > > > > > > > > (MKL)
> > > > > > > > > > > - Git submodules
> > > > > > > > > > >     - cub <https://github.com/dmlc/cub>
> > > > > > > > > > >     - dlpack <https://github.com/dmlc/dlpack>
> > > > > > > > > > >     - dmlc-core <https://github.com/dmlc/dmlc-core>
> > > > > > > > > > >     - googletest <https://github.com/abseil/googletest
> >
> > > > > > > > > > >     - mkldnn <https://github.com/intel/mkl-dnn>
> > > > > > > > > > >     - mshadow <https://github.com/dmlc/mshadow>
> > > > > > > > > > >     - onnx-tensorrt <
> > https://github.com/onnx/onnx-tensorrt
> > > >
> > > > > > > > > > >     - openmp <https://github.com/llvm-mirror/openmp>
> > > > > > > > > > >     - ps-lite <https://github.com/dmlc/ps-lite>
> > > > > > > > > > >     - tvm <https://github.com/dmlc/tvm>
> > > > > > > > > > >
> > > > > > > > > > > this appears to be very heterogeneous
and hard to
> > > > > manage/maintain,
> > > > > > > as
> > > > > > > > > > > multiple various commands are in use
to achieve
> > > dependencies
> > > > > > > > > > installation,
> > > > > > > > > > > as well as multiple places are to look
for dependency
> > > > versions
> > > > > and
> > > > > > > > their
> > > > > > > > > > > updates.
> > > > > > > > > > >
> > > > > > > > > > > with conan, it may became much more
straightforward, as
> > > > > > > dependencies
> > > > > > > > will
> > > > > > > > > > > be declared in single place (conanfile)
and installed
> via
> > > > > single
> > > > > > > > command
> > > > > > > > > > > (conan install).
> > > > > > > > > > >
> > > > > > > > > > > as project is very complex, and has
lots of
> dependencies,
> > > for
> > > > > the
> > > > > > > > first
> > > > > > > > > > > prototype I've used only very few of
dependencies from
> > > conan:
> > > > > > > OpenCV
> > > > > > > > > > > <
> > https://bintray.com/conan-community/conan/opencv%3Aconan
> > > >,
> > > > > > > OpenBLAS
> > > > > > > > > > > <
> > > https://bintray.com/conan-community/conan/openblas%3Aconan>
> > > > > and
> > > > > > > > lapack
> > > > > > > > > > > <
> > https://bintray.com/conan-community/conan/lapack%3Aconan
> > > >.
> > > > > > > > > > > others may be easily added then one
by one, but they
> > first
> > > > has
> > > > > to
> > > > > > > be
> > > > > > > > > > > packaged (not all of them are packaged
yet, e.g.
> > GoogleTest
> > > > > > > > > > > <
> > > > > https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters>
> > > > > > > > is
> > > > > > > > > > > available, while MKL is not).
> > > > > > > > > > >
> > > > > > > > > > > I attach patch which adds an initial
conan support as
> > proof
> > > > of
> > > > > > > > concept.
> > > > > > > > > > > also, I attach two simple build scripts,
which I've
> used
> > to
> > > > > test
> > > > > > > (for
> > > > > > > > > > > Windows and Linux / Mac OS X). Google
Mail blocks .sh
> and
> > > > .cmd
> > > > > > > > > > extensions,
> > > > > > > > > > > so you'll need to rename files.
> > > > > > > > > > > lemme know if you have any further
questions.
> > > > > > > > > > >
> > > > > > > > > > > yours sincerely, Konstantin
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message