mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junru Shao <junrushao1...@gmail.com>
Subject Re: [VOTE] add conan support for Apache MXNet (incubating)
Date Fri, 03 May 2019 17:54:07 GMT
Hi Konstantin, Kellen,

Thank you guys for the very detailed explanation! I was lacking some
relevant contexts and previous discussions, which got me confused
previously.

My understanding that C++ is short of a perfect package manager. I totally
agree that for any C++ project, it would be great to try possible
opportunities to make our building experience smoother. This will largely
help improve the usability of any project in the C++ community.

TensorFlow uses Bazel officially, CMake supported unofficially. PyTorch
seems to use CMake as wellMy. My understanding is that it helps users from
the community to quickly pick up the building process. If we could help
customers (industrial or academic, internal or external) build stuff more
smoothly, it would be definitely a big plus.

It will be a good idea if we have something to try out or play with. For
example, instructions for building MXNet with Conan, customizing MXNet
build with Conan, etc. Considering mshadow is going to be merged into
mxnet, I believe it is a good opportunity to illustrate the power and
convenience of Conan.

Again, thank you guys for the valuable and patient explanation!

Thanks,
Junru

On Fri, May 3, 2019 at 10:14 AM Konstantin Ivlev <tomskside@gmail.com>
wrote:

> Hi Junru,
>
> > I am actually a bit concerned about the security issues. We are asked to
> download binaries from third-party websites, which are not controlled or
> validated by Apache
> it's possible to run conan server inside apache network and download
> binaries and sources only from this remote
>
> > CMake does support download artifacts from a pre-defined website very
> well
> same for conan, it also supports downloading of artifacts from pre-defined
> web-sites
>
> > rather than broken links to nonsense
> please let me know which links are broken. I've checked them all, and they
> are all available for me. may be your ISP is blocking something?
>
> > most of the dependencies you mentioned adopts CMake, rather than Conan
> I don't think it's relevant to compare CMake (meta-build-system) and Conan
> (package manager), they have different purposes, and they might be used
> simultaneously, or only one of them might be used, they aren't mutually
> exclusive.
>
> > Would you mind at least mentioning the benefits somewhere in this thread
> okay, let me list some benefits (not all of them, but something I have on
> mind)
> - cross-platform, runs on Windows, Linux, OSX, FreeBSD, etc., also supports
> cross-compiling for various platforms, like iOS, Android, Emscripten,
> Windows Phone, etc.
> - open-source (MIT licensed) and freeware
> - supports both build from sources and pre-built artifacts (for instance,
> if compare to usage of submodules, you probably wouldn't use pre-built
> binaries in that case, as repo size will grow)
> - supports multiple versions of libraries (some package managers provide
> only "latest")
> - supports options of libraries (whatever you need is configurable per
> library, e.g. shared vs static, with or without assembler, multi-threaded
> vs single-threaded, etc)
> - supports installation of build tools as well (something like bison, flex,
> nasm, yasm, etc.)
> - has concept of profiles and their management (include options, tools and
> environment variables to be used for build)
> - supports various build systems (CMake, Meson, MSBuild, boost build,
> etc.), pretty much build system agnostic
> - decentralized, you may use in-house server (e.g. conan server or
> artifactory), or bintray
> - extensible via hooks
> there are some comparisons of package managers on reddit (e.g.
> https://www.reddit.com/r/cpp/comments/9m4l0p/conan_vcpkg_or_build2/ and
>
> https://www.reddit.com/r/cpp/comments/8ldhu0/opinions_on_the_conan_package_manager/
> ).
>
> > and eagerly asking for avote? I believe that reasonable discussion would
> keep us within a *healthy* discussion.
> I am asking for a vote because I was told to do this on GitHub discussion
> by MXNet developers. I've already started discussions using all suggested
> channels (GitHub, JIRA and this mail list). all questions were answered,
> and no further questions appeared until today. I was thinking we already
> have passed the discussion stage, as all discussion have stalled with no
> objections (nobody clearly said something like "we're going to adopt conan"
> or "we will not use conan definetely"). so my impression was discussion
> stage was already passed, and now it's time for the decision. sorry if I
> had wrong impression, I am not really very familiar with your processes.
>
> > why not I simply specify versions in git submidule, which everyone
> understands how it behaves
> I haven't said usage of submodules itself is something bad, if you're fine
> with submodules, it's okay, but it seems like for MXNet they don't fit all
> use-cases, as some dependencies are downloaded via other ways (as mentioned
> above, via conda, cmake or just downloaded from GitHub archives in CI
> scripts). this causes fragmentation and includes complexity. so, it's clear
> that submodules somehow didn't satisfy all MXNet needs, as they aren't used
> for everything.
>
> > Everyone know how to include a sub-directory in cmake in one line
> probably, because not all of your dependencies are using CMake to build, so
> you can't simply include them into cmake in one line.
>
> yours sincerely, Konstantin
>
> пт, 3 мая 2019 г. в 23:10, Junru Shao <junrushao1994@gmail.com>:
>
> > I am actually a bit concerned about the security issues. We are asked to
> > download binaries from third-party websites, which are not controlled or
> > validated by Apache. Although it is claimed to be “decentralized”, I am
> > really not convinced where the security comes from.
> >
> > In the meantime, sacrificing security doesn’t really bring us tangible
> > benefits. CMake does support download artifacts from a pre-defined
> website
> > very well. We may also have pre-built binaries stored in our CI docker
> > without having to download it them from the internet.
> >
> > Another point is that I am not convinced of any advantage of Conan over
> > other package managers for C++. Would you mind at least mentioning the
> > benefits somewhere in this thread, rather than carelessly includes tons
> of
> > irrelevant links (some of which are even wrong) and eagerly asking for a
> > vote? I believe that reasonable discussion would keep us within a
> *healthy*
> > discussion.
> >
> > Last but not least, as we all know, most of the dependencies you
> mentioned
> > adopts CMake, rather than Conan, the meta-generator which generates
> CMake.
> > I didn’t see your logic stands like “oh you have tons of dependencies so
> > you must use Conan”, why not I simply specify versions in git submidule,
> > which everyone understands how it behaves? Everyone know how to include a
> > sub-directory in cmake in one line, so why we have to write python to
> make
> > it less understandable and more complicated?
> >
> > In conclusion, we need to be reasonable in healthy discussion. I don’t
> > particularly want to rudely +1 or -1 for a thing that is unclear to me,
> but
> > I really want to see pros and cons, discussion about issues and concerns,
> > rather than broken links to nonsense.
> >
> > Looking forward to your reply!
> >
> > Thanks,
> > Junru
> >
> > On Fri, May 3, 2019 at 08:05 kellen sunderland <
> > kellen.sunderland@gmail.com>
> > wrote:
> >
> > > Hey Konstantin.  Thanks for starting an email thread and sorry for the
> > > confusion.  I think the ides is that we should discuss and agree on
> > > Conan.io adoption first on the dev list, then start merging PRs.
> Release
> > > 1.4.1 is already in testing and the 1.5 code freeze deadline is also
> near
> > > so I think it could be difficult to make such a large change on one of
> > > those releases.  I've looked into package management solutions for the
> > > project before.  I was in favour of hunter, but I think Conan's
> adoption
> > > rate now makes it the best option.  It's simple to use and is becoming
> > > industry standard, with a minor downside of requiring Python (which has
> > > meanwhile become the most popular dev language).  I'd personally be -1
> > for
> > > 1.4.1 or 1.5, +1 for using Conan in 1.6 or 2.0.
> > >
> > > -Kellen
> > >
> > > On Fri, May 3, 2019 at 12:59 AM Konstantin Ivlev <tomskside@gmail.com>
> > > wrote:
> > >
> > > > hi Sheng Zha,
> > > >
> > > > on pull request review I was told by Anirudh anirudhacharya and
> Roshani
> > > > Nagmote to start discussion/vote on the mxnet dev list. it seems to
> be
> > a
> > > > vicious circle now - on GitHub I am told to use vote, and on vote I
> am
> > > told
> > > > to use GitHub, this doesn't help much.
> > > > FYI GitHub review stuck, it's already opened since November 2018, and
> > > it's
> > > > still not approved (however, there were no objections during the
> > review).
> > > > Previous discussion in e-mail thread also didn't encounter any
> > > objections,
> > > > and all questions were answered.
> > > > JIRA ticket has no discussion at all (except it has duplicates of
> > > comments
> > > > made on GitHub).
> > > > so let's process with 3-day vote for now, as other communication
> > channels
> > > > were already tried with no success.
> > > >
> > > > yours sincerely, Konstantin
> > > >
> > > > пт, 3 мая 2019 г. в 14:17, Sheng Zha <zhasheng@apache.org>:
> > > >
> > > > > Hi Konstantin,
> > > > >
> > > > > While conan looks like an option that's worth exploring, given that
> > > your
> > > > > request is to merge the pull request, I'd suggest that the request
> > > should
> > > > > go through the regular pull request review and it doesn't really
> > need a
> > > > > vote (as it doesn't substitute reviews anyway)
> > > > >
> > > > > If you would like to gather more attention to it, feel free to ping
> > in
> > > a
> > > > > discussion thread.
> > > > >
> > > > > -sz
> > > > >
> > > > > On 2019/05/03 06:29:55, Konstantin Ivlev <tomskside@gmail.com>
> > wrote:
> > > > > > Dear MXNet community,
> > > > > >
> > > > > > This is the 3-day vote to add conan support for Apache MXNet
> > > > (incubating)
> > > > > > version v1.4.1.
> > > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and
> close
> > > on
> > > > > May
> > > > > > 06 23:59:59.
> > > > > >
> > > > > > Background: conan is open-source, freeware, cross-platform
> package
> > > > > manager
> > > > > > for C and C++ projects, written in python. it provides
> integration
> > > with
> > > > > > various build systems, include CMake. conan may use bintray
as a
> > > server
> > > > > to
> > > > > > store and download pre-built packages, or packages might be
> always
> > > > built
> > > > > > from sources.
> > > > > >
> > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating)
is
> > > using
> > > > > > several ways to fetch 3rd-party dependencies simultaneously,
for
> > > > > instance:
> > > > > > 1. download GitHub archives during the build
> > > > > > - OpenBLAS
> > > > > > - OpenCV
> > > > > > 2. conda (alternative way to GitHub archives)
> > > > > > 3. download from CMake
> > > > > > - Intel Math Kernel Library (MKL)
> > > > > > 4. Git submodules
> > > > > > - cub
> > > > > > - dlpack
> > > > > > - dmlc-core
> > > > > > - googletest
> > > > > > - mkldnn
> > > > > > - mshadow
> > > > > > - onnx-tensorrt
> > > > > > - openmp
> > > > > > - ps-lite
> > > > > > - tvm
> > > > > > therefore, there are multiple places to look for 3rd parties,
and
> > its
> > > > > hard
> > > > > > to update them, as you need to remember or figure it out how
to
> > > update
> > > > a
> > > > > > particular dependency to newer version, for instance.
> > > > > > current Apache MXNet (incubating) build instructions differ
very
> > much
> > > > per
> > > > > > platform, require to download and unzip some archives manually,
> > > > > specifying
> > > > > > variables with paths to this archives, in conjunction of updating
> > git
> > > > > > submodules,
> > > > > >
> > > > > > Action: merge pull request providing an initial conan support
for
> > > > Apache
> > > > > > MXNet (incubating). support conan as an alternate approach to
> fetch
> > > > > various
> > > > > > 3rd-party dependencies. old approaches will be still available,
> > > > supported
> > > > > > and left intact.
> > > > > >
> > > > > > Below are links to
> > > > > > 1) conan web-site:  https://conan.io/
> > > > > > 2) conan GitHub repository: https://github.com/conan-io/conan
> > > > > > 3) conan documentation: https://docs.conan.io/en/latest/
> > > > > > 4) bintray: https://bintray.com
> > > > > > 5) pull request adding conan support to Apache MXNet
> (incubating):
> > > > > > https://github.com/apache/incubator-mxnet/pull/13400
> > > > > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229
> > > > > > 7) previous email discussion:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E
> > > > > > 8) MXNet build instructions:
> > > > > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html
> > > > > > 9) MXNet build instructions (Windows):
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html
> > > > > > 10) MXNet build instructions (OSX):
> > > > > >
> > > >
> > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html
> > > > > > 11) MXNet build instructions (Linux):
> > > > > >
> > > > >
> > > >
> > >
> >
> http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html
> > > > > > 12) MXNet development setup (OSX):
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac
> > > > > >
> > > > > > Please remember to TEST first before voting accordingly:
> > > > > > +1 = approve
> > > > > > +0 = no opinion
> > > > > > -1 = disapprove (provide reason)
> > > > > >
> > > > > > Best regards,
> > > > > > Konstantin Ivlev
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message