mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Ivlev <tomsks...@gmail.com>
Subject Re: [VOTE] add conan support for Apache MXNet (incubating)
Date Mon, 06 May 2019 08:10:29 GMT
Hi Sheng Zha,

>  Currently, the linked PR only includes OpenBLAS
actually, it's OpenBLAS + OpenCV + lapack, three libraries as
proof-of-concept

> A proof-of-concept that shows it actually replaces more dependency than
openblas would be helpful
may we define an actual scope then, e.g. how much dependencies - 2, 3, 5,
half of them or all of them?

> If the value proposition of conan is to simplify the dependency
management, then it should unify other solutions instead of keeping these
solutions around.
personally, I don't think it's easy to do in single shot. I personally
prefer small incremental changes, and starting with some MVP, where M is
minimal, meaning least possible effort (e.g. number of dependencies) used
from conan. then eventually migrate other dependencies to conan one by one,
until it fully migrates. but that's just my personal vision, if you see
that this strategy is wrong, it's okay.

- It's unclear how it impacts people with only an archive of mxnet source
code but without network access.
as I have tried, conan doesn't change that much. if you have no network
access, and only mxnet source code, then "git submodule init" will fail for
you. if you have also archives of dependencies somewhere on your
hard-drive, you can manage submodules to clone from the local repository
rather GitHub. but then you will stuck at the CMake generation step, which
will fail to download Intel MKL.
if you use conan instead of submodules and CMake download, there is not
much difference - it will fail to fetch same dependencies without internet
connection on clean machine. if you somehow happen to have ab archive of
conan cache on your hard-drive, you may unpack it and use without internet
connection.
but, I am not sure, for me it looks like strange use-case. I can imagine
you have developers with no internet connection and send them parcels with
CDs of mxnet source code, but I think it's hard to manage such workflow
with any tool, no matter submodules, CMake download or conan.

yours sincerely, Konstantin


вс, 5 мая 2019 г. в 06:25, Sheng Zha <zhasheng@apache.org>:

> To be clear, my intention is really to prevent a seemingly good solution
> to exacerbate the problem that it sets out to solve. This tends to happen
> when there are not enough people to drive it to the end.
>
> If there are additional values in this solution that people feel outweighs
> the problems below, I'd be more than happy to be persuaded to vote
> otherwise.
>
> -sz
>
> On 2019/05/04 23:08:43, Sheng Zha <zhasheng@apache.org> wrote:
> > Thank you for the explanation and sorry that I missed the earlier
> context as it has been a while. While I like the idea of simplifying the
> dependency management with tools like conan, I have the following concerns
> on this vote as-is (it's also my take on why I think the PR is stuck):
> >
> > - It's unclear how much dependency needs can conan help in mxnet builds.
> >   Currently, the linked PR only includes OpenBLAS. A proof-of-concept
> that shows it actually replaces more dependency than openblas would be
> helpful. On a high-level, there are two types of builds for mxnet:
> >   * User's custom build-from-source: 1) usually dynamic linking is used.
> 2) users may not enable all features, and users may want to pull a subset
> of the dependencies. 3) users may want mxnet build system to pull the
> dependencies, or they may not. (for conan it's ok to just focus on the
> former)
> >   * Binary distribution for pip and maven: 1) static linking is used
> (requires -fPIC). 2) all features are enabled. 3) dependencies are pulled
> in with scripts in mxnet.
> >   Handling one of the above cases would be a good showcase for the value
> of the new tool.
> >
> > - It's unclear how it impacts people with only an archive of mxnet
> source code but without network access.
> >   This applies for the dependencies that are captured as submodules that
> you mentioned as a way that mxnet manages dependency.
> >
> > - If the value proposition of conan is to simplify the dependency
> management, then it should unify other solutions instead of keeping these
> solutions around.
> >
> > Overall, it would be helpful to have a clear message such as what
> exactly conan can replace, and having a proof of concept that works for
> this would be helpful. Otherwise, I fear that we may be introducing yet
> another way to manage dependency that further complicates the existing
> problem.
> >
> > That said, I'm not suggesting that we impose the burden to implement
> everything on you alone, and it's ok to rally people who are interested in
> this solution to help out. To facilitate this, I created a feature branch
> so that it's easier for you and people who are enthusiastic about this to
> work together [1].
> >
> > For now, I'm voting -1 to this proposal and I hope you understand.
> >
> > -sz
> >
> > [1] https://github.com/apache/incubator-mxnet/tree/conan
> >
> > On 2019/05/03 07:51:34, Konstantin Ivlev <tomskside@gmail.com> wrote:
> > > hi Sheng Zha,
> > >
> > > on pull request review I was told by Anirudh anirudhacharya and Roshani
> > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be
> a
> > > vicious circle now - on GitHub I am told to use vote, and on vote I am
> told
> > > to use GitHub, this doesn't help much.
> > > FYI GitHub review stuck, it's already opened since November 2018, and
> it's
> > > still not approved (however, there were no objections during the
> review).
> > > Previous discussion in e-mail thread also didn't encounter any
> objections,
> > > and all questions were answered.
> > > JIRA ticket has no discussion at all (except it has duplicates of
> comments
> > > made on GitHub).
> > > so let's process with 3-day vote for now, as other communication
> channels
> > > were already tried with no success.
> > >
> > > yours sincerely, Konstantin
> > >
> > > пт, 3 мая 2019 г. в 14:17, Sheng Zha <zhasheng@apache.org>:
> > >
> > > > Hi Konstantin,
> > > >
> > > > While conan looks like an option that's worth exploring, given that
> your
> > > > request is to merge the pull request, I'd suggest that the request
> should
> > > > go through the regular pull request review and it doesn't really
> need a
> > > > vote (as it doesn't substitute reviews anyway)
> > > >
> > > > If you would like to gather more attention to it, feel free to ping
> in a
> > > > discussion thread.
> > > >
> > > > -sz
> > > >
> > > > On 2019/05/03 06:29:55, Konstantin Ivlev <tomskside@gmail.com>
> wrote:
> > > > > Dear MXNet community,
> > > > >
> > > > > This is the 3-day vote to add conan support for Apache MXNet
> (incubating)
> > > > > version v1.4.1.
> > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and
> close on
> > > > May
> > > > > 06 23:59:59.
> > > > >
> > > > > Background: conan is open-source, freeware, cross-platform package
> > > > manager
> > > > > for C and C++ projects, written in python. it provides integration
> with
> > > > > various build systems, include CMake. conan may use bintray as a
> server
> > > > to
> > > > > store and download pre-built packages, or packages might be always
> built
> > > > > from sources.
> > > > >
> > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is
> using
> > > > > several ways to fetch 3rd-party dependencies simultaneously, for
> > > > instance:
> > > > > 1. download GitHub archives during the build
> > > > > - OpenBLAS
> > > > > - OpenCV
> > > > > 2. conda (alternative way to GitHub archives)
> > > > > 3. download from CMake
> > > > > - Intel Math Kernel Library (MKL)
> > > > > 4. Git submodules
> > > > > - cub
> > > > > - dlpack
> > > > > - dmlc-core
> > > > > - googletest
> > > > > - mkldnn
> > > > > - mshadow
> > > > > - onnx-tensorrt
> > > > > - openmp
> > > > > - ps-lite
> > > > > - tvm
> > > > > therefore, there are multiple places to look for 3rd parties, and
> its
> > > > hard
> > > > > to update them, as you need to remember or figure it out how to
> update a
> > > > > particular dependency to newer version, for instance.
> > > > > current Apache MXNet (incubating) build instructions differ very
> much per
> > > > > platform, require to download and unzip some archives manually,
> > > > specifying
> > > > > variables with paths to this archives, in conjunction of updating
> git
> > > > > submodules,
> > > > >
> > > > > Action: merge pull request providing an initial conan support for
> Apache
> > > > > MXNet (incubating). support conan as an alternate approach to fetch
> > > > various
> > > > > 3rd-party dependencies. old approaches will be still available,
> supported
> > > > > and left intact.
> > > > >
> > > > > Below are links to
> > > > > 1) conan web-site:  https://conan.io/
> > > > > 2) conan GitHub repository: https://github.com/conan-io/conan
> > > > > 3) conan documentation: https://docs.conan.io/en/latest/
> > > > > 4) bintray: https://bintray.com
> > > > > 5) pull request adding conan support to Apache MXNet (incubating):
> > > > > https://github.com/apache/incubator-mxnet/pull/13400
> > > > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229
> > > > > 7) previous email discussion:
> > > > >
> > > >
> https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E
> > > > > 8) MXNet build instructions:
> > > > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html
> > > > > 9) MXNet build instructions (Windows):
> > > > >
> > > >
> https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html
> > > > > 10) MXNet build instructions (OSX):
> > > > >
> http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html
> > > > > 11) MXNet build instructions (Linux):
> > > > >
> > > >
> http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html
> > > > > 12) MXNet development setup (OSX):
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac
> > > > >
> > > > > Please remember to TEST first before voting accordingly:
> > > > > +1 = approve
> > > > > +0 = no opinion
> > > > > -1 = disapprove (provide reason)
> > > > >
> > > > > Best regards,
> > > > > Konstantin Ivlev
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message