mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Chernov <mecher...@gmail.com>
Subject Re: using conan to manage Apache Incubator MXNet project dependencies
Date Wed, 05 Dec 2018 21:06:55 GMT
What I rather meant is a 'sandboxed build', for example without internet
connection. Would this be possible?

Or is it possible to define and maintain own conan package registry?

ср, 28 нояб. 2018 г. в 16:33, Konstantin Ivlev <tomskside@gmail.com>:

> > Is it possible to have an offline build somehow? For example, if all
> > dependencies would be stored locally. Probably, that would require some
> > modifications to the conan file, right?
> as soon as all dependencies are stored locally,
> conan will use binaries from the local cache.
> no modifications are required to conan file or workflow in this case.
>
> ср, 28 нояб. 2018 г. в 22:24, Anton Chernov <mechernov@gmail.com>:
>
> > Great, thank you for your answers.
> >
> > Is it possible to have an offline build somehow? For example, if all
> > dependencies would be stored locally. Probably, that would require some
> > modifications to the conan file, right?
> >
> >
> > вт, 27 нояб. 2018 г. в 16:59, Konstantin Ivlev <tomskside@gmail.com>:
> >
> > > >  Would it be possible to define cmake itself as a conan dependency
> for
> > > MXNet?
> > > yes, it is definitely possible
> > > for instance, to declare CMake 3.13.0 as a dependency, the following
> line
> > > has to be added to the conanfile.py:
> > >
> > > build_requires = "cmake_installer/3.13.0@conan/stable"
> > >
> > > other build tools might be added in a similar manner.
> > >
> > > вт, 27 нояб. 2018 г. в 22:54, Anton Chernov <mechernov@gmail.com>:
> > >
> > > > I think I asked this already, but want to confirm in regards to the
> > > > following discussion:
> > > >
> > > > MXNet CMake build - raise minimal required version
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/09772f9447ebff72cdfea5512ae70e295db8a21ba0ddb39359cd0a77@%3Cdev.mxnet.apache.org%3E
> > > >
> > > > Would it be possible to define cmake itself as a conan dependency for
> > > > MXNet?
> > > >
> > > > Best
> > > > Anton
> > > >
> > > > вт, 27 нояб. 2018 г. в 15:44, Konstantin Ivlev <tomskside@gmail.com
> >:
> > > >
> > > > > both questions are totally valid.
> > > > > > Is it easy to create a build which will build dependencies from
> > > source?
> > > > > 1. yep, it's very easy, just add `--build` argument to the `conan
> > > > install`
> > > > > command line
> > > > > >  What guarantees you get with conan with regards to ABI / C++
> > stdlib
> > > > > binary compatibility of the pulled dependencies?
> > > > > 2. the default binary compatibility model may be found in
> > settings.yml
> > > (
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/conan-io/conan/blob/develop/conans/client/conf/__init__.py#L14
> > > > > )
> > > > > it includes the operation system, architecture, compiler, its
> version
> > > and
> > > > > C++ stdlib.
> > > > > while it's true that it won't cover 100% of use-cases (e.g. for
> older
> > > > > distros or older glibc), it's very good default for most users.
> > > > > you may always define custom binary compatibility attributes in
> your
> > > > > settings.yml (e.g. distro, microarchitecture, sanitizers, etc.).
> > > > >
> > > > >
> > > > > вт, 27 нояб. 2018 г. в 21:17, Pedro Larroy <
> > > pedro.larroy.lists@gmail.com
> > > > >:
> > > > >
> > > > > > Thanks both for the detailed explanations. Couple of more
> > questions:
> > > > > >
> > > > > > Is it easy to create a build which will build dependencies from
> > > source?
> > > > > > What guarantees you get with conan with regards to ABI / C++
> stdlib
> > > > > > binary compatibility of the pulled dependencies?
> > > > > >
> > > > > > Just to clarify: My concerns are in terms of reproducible builds
> /
> > > > > > source only distribution and undefined behaviour due to different
> > > > > > compiler / stdlib versions. Are these valid or is it oudated
> > > > > > knowledge?
> > > > > >
> > > > > > Pedro.
> > > > > > On Tue, Nov 27, 2018 at 2:34 PM Diego Rodriguez-Losada
> > > > > > <diego.rlosada@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Pedro,
> > > > > > >
> > > > > > > Conan is distributed. So besides building from sources
the
> > > > > dependencies,
> > > > > > it
> > > > > > > is also possible to create binaries yourself for those
> > dependencies
> > > > > (with
> > > > > > > the existing recipes, or your own recipes), and host them
in
> your
> > > own
> > > > > > repo
> > > > > > > (Bintray OSS repo, or Artifactory).
> > > > > > >
> > > > > > > This will provide both the security that you own the
> dependencies
> > > > > > binaries
> > > > > > > and the convenience and speed of not having to build from
> > sources.
> > > > Even
> > > > > > if
> > > > > > > you provide the binaries, consumers can always fallback
to
> build
> > > from
> > > > > > > sources too.
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Diego
> > > > > > >
> > > > > > > El mar., 27 nov. 2018 a las 13:34, Konstantin Ivlev (<
> > > > > > tomskside@gmail.com>)
> > > > > > > escribió:
> > > > > > >
> > > > > > > > Hi Pedro,
> > > > > > > >
> > > > > > > > yes, you're absolutely right, by default, conan will
be
> pulling
> > > > > > prebuilt
> > > > > > > > binaries for the libraries from the bintray.
> > > > > > > > however, if prebuilt binaries are not available (e.g.
because
> > you
> > > > use
> > > > > > some
> > > > > > > > different compiler for which we don't have prebuilt
> binaries),
> > > > > > > > or if you want to build binaries yourself for some
another
> > > reason,
> > > > > > > > then libraries always might be built from source (by
passing
> > e.g.
> > > > > > "--build
> > > > > > > > always", "--build missing" or "--build <library>"
to the
> conan
> > > > > install
> > > > > > > > command line).
> > > > > > > >
> > > > > > > > yours sincerely, Konstantin
> > > > > > > >
> > > > > > > > вт, 27 нояб. 2018 г. в 19:27, Pedro Larroy
<
> > > > > > pedro.larroy.lists@gmail.com>:
> > > > > > > >
> > > > > > > > > Hi Konstantin
> > > > > > > > >
> > > > > > > > > Thanks for this contribution. With your proposed
changes,
> > when
> > > > > > > > > building MXNet we will be pulling binaries for
the
> libraries
> > > > > managed
> > > > > > > > > by conan?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Pedro.
> > > > > > > > > On Mon, Nov 26, 2018 at 11:43 AM Konstantin Ivlev
<
> > > > > > tomskside@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hello, Ivan,
> > > > > > > > > >
> > > > > > > > > > could you possibly clarify your question
(may be
> explaining
> > > the
> > > > > > > > use-case
> > > > > > > > > > behind it)?
> > > > > > > > > > Gradle appears to be build system, AFAIK
more popular in
> > Java
> > > > > > world.
> > > > > > > > > > meanwhile, Apache Incubator MXNet project
uses CMake as
> its
> > > > build
> > > > > > > > system.
> > > > > > > > > > please correct me, I am wrong.
> > > > > > > > > > in general, conan, as a package manager,
is pretty
> > > > > > > > build-system-agnostic,
> > > > > > > > > > and it may work with arbitrary build systems
(to count a
> > few,
> > > > > > CMake,
> > > > > > > > > > premake, qmake, full list:
> > > > > > > > > >
> https://docs.conan.io/en/latest/reference/generators.html
> > ).
> > > I
> > > > > > don't
> > > > > > > > > think
> > > > > > > > > > Gradle is exception here.
> > > > > > > > > > also, for instance, Android Studio also
uses Gradle for
> > > Android
> > > > > C++
> > > > > > > > > > projects, and conan works flawlessly in
this particular
> > case
> > > (
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://blog.conan.io/2018/02/13/Android-Studio-project-Conan-Boost.html
> > > > > > > > > ).
> > > > > > > > > >
> > > > > > > > > > yours sincerely, Konstantin
> > > > > > > > > >
> > > > > > > > > > пн, 26 нояб. 2018 г. в 16:43, Ivan
Serdyuk <
> > > > > > > > local.tourist.kiev@gmail.com
> > > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Kostantin, and what (overall) option
with using Gradle?
> > > Does
> > > > > this
> > > > > > > > your
> > > > > > > > > > > suggested package manager has been
supported by Gradle?
> > > > > > > > > > >
> > > > > > > > > > > Ivan
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 26, 2018 at 9:43 AM Konstantin
Ivlev <
> > > > > > > > tomskside@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > hello,
> > > > > > > > > > > >
> > > > > > > > > > > > this email is related to the following
PR and JIRA
> > > ticket:
> > > > > > > > > > > > - [MXNET-1229] use OpenBLAS, lapack
& OpenCV from
> conan
> > > > > > > > > > > > <
> https://github.com/apache/incubator-mxnet/pull/13400>
> > > > > > > > > > > > - use conan to manage project
dependencies
> > > > > > > > > > > > <https://issues.apache.org/jira/browse/MXNET-1229>
> > > > > > > > > > > >
> > > > > > > > > > > > conan <https://conan.io>
is an open-source package
> > > manager
> > > > > > for C++
> > > > > > > > > > > > projects. it allows to manage
project dependencies in
> > > > > > transparent
> > > > > > > > and
> > > > > > > > > > > > declarative manner.
> > > > > > > > > > > >
> > > > > > > > > > > > currently, apache incubator-mxnet
project uses the
> > > > following
> > > > > > > > > different
> > > > > > > > > > > > ways to manage its dependencies:
> > > > > > > > > > > >
> > > > > > > > > > > > - download GitHub archives during
the build
> > > > > > > > > > > >     - OpenBLAS <https://github.com/xianyi/OpenBLAS>
> > > > > > > > > > > >     - OpenCV <https://github.com/opencv/opencv>
> > > > > > > > > > > > - conda <https://conda.io/>
(alternative way to
> GitHub
> > > > > > archives)
> > > > > > > > > > > > - download from CMake
> > > > > > > > > > > >     - Intel Math Kernel Library
<
> > > > > > > > > https://software.intel.com/en-us/mkl>
> > > > > > > > > > > > (MKL)
> > > > > > > > > > > > - Git submodules
> > > > > > > > > > > >     - cub <https://github.com/dmlc/cub>
> > > > > > > > > > > >     - dlpack <https://github.com/dmlc/dlpack>
> > > > > > > > > > > >     - dmlc-core <https://github.com/dmlc/dmlc-core>
> > > > > > > > > > > >     - googletest <
> https://github.com/abseil/googletest
> > >
> > > > > > > > > > > >     - mkldnn <https://github.com/intel/mkl-dnn>
> > > > > > > > > > > >     - mshadow <https://github.com/dmlc/mshadow>
> > > > > > > > > > > >     - onnx-tensorrt <
> > > https://github.com/onnx/onnx-tensorrt
> > > > >
> > > > > > > > > > > >     - openmp <https://github.com/llvm-mirror/openmp>
> > > > > > > > > > > >     - ps-lite <https://github.com/dmlc/ps-lite>
> > > > > > > > > > > >     - tvm <https://github.com/dmlc/tvm>
> > > > > > > > > > > >
> > > > > > > > > > > > this appears to be very heterogeneous
and hard to
> > > > > > manage/maintain,
> > > > > > > > as
> > > > > > > > > > > > multiple various commands are
in use to achieve
> > > > dependencies
> > > > > > > > > > > installation,
> > > > > > > > > > > > as well as multiple places are
to look for dependency
> > > > > versions
> > > > > > and
> > > > > > > > > their
> > > > > > > > > > > > updates.
> > > > > > > > > > > >
> > > > > > > > > > > > with conan, it may became much
more straightforward,
> as
> > > > > > > > dependencies
> > > > > > > > > will
> > > > > > > > > > > > be declared in single place (conanfile)
and installed
> > via
> > > > > > single
> > > > > > > > > command
> > > > > > > > > > > > (conan install).
> > > > > > > > > > > >
> > > > > > > > > > > > as project is very complex, and
has lots of
> > dependencies,
> > > > for
> > > > > > the
> > > > > > > > > first
> > > > > > > > > > > > prototype I've used only very
few of dependencies
> from
> > > > conan:
> > > > > > > > OpenCV
> > > > > > > > > > > > <
> > > https://bintray.com/conan-community/conan/opencv%3Aconan
> > > > >,
> > > > > > > > OpenBLAS
> > > > > > > > > > > > <
> > > > https://bintray.com/conan-community/conan/openblas%3Aconan>
> > > > > > and
> > > > > > > > > lapack
> > > > > > > > > > > > <
> > > https://bintray.com/conan-community/conan/lapack%3Aconan
> > > > >.
> > > > > > > > > > > > others may be easily added then
one by one, but they
> > > first
> > > > > has
> > > > > > to
> > > > > > > > be
> > > > > > > > > > > > packaged (not all of them are
packaged yet, e.g.
> > > GoogleTest
> > > > > > > > > > > > <
> > > > > > https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters
> >
> > > > > > > > > is
> > > > > > > > > > > > available, while MKL is not).
> > > > > > > > > > > >
> > > > > > > > > > > > I attach patch which adds an initial
conan support as
> > > proof
> > > > > of
> > > > > > > > > concept.
> > > > > > > > > > > > also, I attach two simple build
scripts, which I've
> > used
> > > to
> > > > > > test
> > > > > > > > (for
> > > > > > > > > > > > Windows and Linux / Mac OS X).
Google Mail blocks .sh
> > and
> > > > > .cmd
> > > > > > > > > > > extensions,
> > > > > > > > > > > > so you'll need to rename files.
> > > > > > > > > > > > lemme know if you have any further
questions.
> > > > > > > > > > > >
> > > > > > > > > > > > yours sincerely, Konstantin
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message