From dev-return-4997-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Wed Nov 28 16:33:49 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 23708180658 for ; Wed, 28 Nov 2018 16:33:48 +0100 (CET) Received: (qmail 98549 invoked by uid 500); 28 Nov 2018 15:33:48 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 98527 invoked by uid 99); 28 Nov 2018 15:33:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Nov 2018 15:33:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1A4D4CFAF8 for ; Wed, 28 Nov 2018 15:33:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.798 X-Spam-Level: ** X-Spam-Status: No, score=2.798 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id IZ1E4s9CIkkh for ; Wed, 28 Nov 2018 15:33:43 +0000 (UTC) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A04EB60DB2 for ; Wed, 28 Nov 2018 15:27:09 +0000 (UTC) Received: by mail-oi1-f171.google.com with SMTP id a77so22880492oii.5 for ; Wed, 28 Nov 2018 07:27:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=1oGY7AcBgbQ8W1haYBTFvRWm5SCbkuCEJDyw/Eeh7YM=; b=J4sq+Isv6nFTAb99h9FtflyLARPU2zz6n/4pqbFjmPvDwsPk4S8lYE+XwJL42p2rE9 EVRyAUbpgvLdfjqOEuTn/3s5XX5Dk9jat3ddDM3InSNvNIVHtLDJ8f1cjRuAxfj8+E/b RuMqRStanXicVjNACMLE8DyuxAWd8Lyuw9oJhPqJziQqzJPx1IbRXZVB26Zq5TrfY9+g lxkmNsGF0kCiollwTlDsi0/Afo/XMgyFb4zR1fm3emg2nOpgE4ElBojQjAaXJ5cn77k7 7QC+rb2aDdIWBUG6pwufnilf9yq7YfWonGyXHaEGL+2YC6MiYX1NWYB5YFsfT4SAPEal ilqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=1oGY7AcBgbQ8W1haYBTFvRWm5SCbkuCEJDyw/Eeh7YM=; b=HtP+tNgdNtr4vbOi2XvzhrCJGjjdNR0mZKDvHM5ZJ8xAR9liuSMUCB149w2P6h1PBU t8a//P44q+vS0TpNLSiRV3fRO6LOFc6xKMUyQsk+8qy5sF1DE4UnVUPABLedKU/bhvvb vGzPRk6W4Q6XLhbdmI+rh4C3slnfkDLgklUs1r4EWdGP02lxJMODtcGZovfB0oEMcS9U VtT8sB4MyEnYAxA5Y+zuTr5jKha/2gdNdHXUiPFG1rmQrJdh272xVToWi8vhH0guoCb9 BkQZvZ7lKZmoXgt6hAllmzNojKFs6c6CJj7BQPLU3HRPBsO02Cs3K3p5iYLbVXJI/eCe mvFQ== X-Gm-Message-State: AGRZ1gIQcMA3qrTbPwYDr6fH513y+lzW/2nvNFWANnaL/fYywXMxp8Mi IE+G/g4+bXw6V4Y7gqBMJ7UVHEv7R+mFSXuL88C4TEos X-Google-Smtp-Source: AJdET5eG/eG7n2vk5DDN538qL8n79BQcTpCsh9gzqDB6CBKd+DvHeWXCK79Eawx+laKLJl9SWqCcMmAtDhj8+2Dl80c= X-Received: by 2002:aca:e346:: with SMTP id a67mr21768327oih.81.1543418828734; Wed, 28 Nov 2018 07:27:08 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Konstantin Ivlev Date: Wed, 28 Nov 2018 22:27:12 +0700 Message-ID: Subject: Re: using conan to manage Apache Incubator MXNet project dependencies To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="00000000000006c610057bbb332e" --00000000000006c610057bbb332e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > Is it possible to have an offline build somehow? For example, if all > dependencies would be stored locally. Probably, that would require some > modifications to the conan file, right? as soon as all dependencies are stored locally, conan will use binaries from the local cache. no modifications are required to conan file or workflow in this case. =D1=81=D1=80, 28 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 22:24, Anton= Chernov : > Great, thank you for your answers. > > Is it possible to have an offline build somehow? For example, if all > dependencies would be stored locally. Probably, that would require some > modifications to the conan file, right? > > > =D0=B2=D1=82, 27 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 16:59, Kon= stantin Ivlev : > > > > Would it be possible to define cmake itself as a conan dependency fo= r > > MXNet? > > yes, it is definitely possible > > for instance, to declare CMake 3.13.0 as a dependency, the following li= ne > > has to be added to the conanfile.py: > > > > build_requires =3D "cmake_installer/3.13.0@conan/stable" > > > > other build tools might be added in a similar manner. > > > > =D0=B2=D1=82, 27 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 22:54, A= nton Chernov : > > > > > I think I asked this already, but want to confirm in regards to the > > > following discussion: > > > > > > MXNet CMake build - raise minimal required version > > > > > > > > > https://lists.apache.org/thread.html/09772f9447ebff72cdfea5512ae70e295db8= a21ba0ddb39359cd0a77@%3Cdev.mxnet.apache.org%3E > > > > > > Would it be possible to define cmake itself as a conan dependency for > > > MXNet? > > > > > > Best > > > Anton > > > > > > =D0=B2=D1=82, 27 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 15:44,= Konstantin Ivlev : > > > > > > > both questions are totally valid. > > > > > Is it easy to create a build which will build dependencies from > > source? > > > > 1. yep, it's very easy, just add `--build` argument to the `conan > > > install` > > > > command line > > > > > What guarantees you get with conan with regards to ABI / C++ > stdlib > > > > binary compatibility of the pulled dependencies? > > > > 2. the default binary compatibility model may be found in > settings.yml > > ( > > > > > > > > > > > > > > https://github.com/conan-io/conan/blob/develop/conans/client/conf/__init_= _.py#L14 > > > > ) > > > > it includes the operation system, architecture, compiler, its versi= on > > and > > > > C++ stdlib. > > > > while it's true that it won't cover 100% of use-cases (e.g. for old= er > > > > distros or older glibc), it's very good default for most users. > > > > you may always define custom binary compatibility attributes in you= r > > > > settings.yml (e.g. distro, microarchitecture, sanitizers, etc.). > > > > > > > > > > > > =D0=B2=D1=82, 27 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 21:1= 7, Pedro Larroy < > > pedro.larroy.lists@gmail.com > > > >: > > > > > > > > > Thanks both for the detailed explanations. Couple of more > questions: > > > > > > > > > > Is it easy to create a build which will build dependencies from > > source? > > > > > What guarantees you get with conan with regards to ABI / C++ stdl= ib > > > > > binary compatibility of the pulled dependencies? > > > > > > > > > > Just to clarify: My concerns are in terms of reproducible builds = / > > > > > source only distribution and undefined behaviour due to different > > > > > compiler / stdlib versions. Are these valid or is it oudated > > > > > knowledge? > > > > > > > > > > Pedro. > > > > > On Tue, Nov 27, 2018 at 2:34 PM Diego Rodriguez-Losada > > > > > wrote: > > > > > > > > > > > > Hi Pedro, > > > > > > > > > > > > Conan is distributed. So besides building from sources the > > > > dependencies, > > > > > it > > > > > > is also possible to create binaries yourself for those > dependencies > > > > (with > > > > > > the existing recipes, or your own recipes), and host them in yo= ur > > own > > > > > repo > > > > > > (Bintray OSS repo, or Artifactory). > > > > > > > > > > > > This will provide both the security that you own the dependenci= es > > > > > binaries > > > > > > and the convenience and speed of not having to build from > sources. > > > Even > > > > > if > > > > > > you provide the binaries, consumers can always fallback to buil= d > > from > > > > > > sources too. > > > > > > > > > > > > Kind regards, > > > > > > Diego > > > > > > > > > > > > El mar., 27 nov. 2018 a las 13:34, Konstantin Ivlev (< > > > > > tomskside@gmail.com>) > > > > > > escribi=C3=B3: > > > > > > > > > > > > > Hi Pedro, > > > > > > > > > > > > > > yes, you're absolutely right, by default, conan will be pulli= ng > > > > > prebuilt > > > > > > > binaries for the libraries from the bintray. > > > > > > > however, if prebuilt binaries are not available (e.g. because > you > > > use > > > > > some > > > > > > > different compiler for which we don't have prebuilt binaries)= , > > > > > > > or if you want to build binaries yourself for some another > > reason, > > > > > > > then libraries always might be built from source (by passing > e.g. > > > > > "--build > > > > > > > always", "--build missing" or "--build " to the cona= n > > > > install > > > > > > > command line). > > > > > > > > > > > > > > yours sincerely, Konstantin > > > > > > > > > > > > > > =D0=B2=D1=82, 27 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0= =B2 19:27, Pedro Larroy < > > > > > pedro.larroy.lists@gmail.com>: > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > > Thanks for this contribution. With your proposed changes, > when > > > > > > > > building MXNet we will be pulling binaries for the librarie= s > > > > managed > > > > > > > > by conan? > > > > > > > > > > > > > > > > > > > > > > > > Pedro. > > > > > > > > On Mon, Nov 26, 2018 at 11:43 AM Konstantin Ivlev < > > > > > tomskside@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hello, Ivan, > > > > > > > > > > > > > > > > > > could you possibly clarify your question (may be explaini= ng > > the > > > > > > > use-case > > > > > > > > > behind it)? > > > > > > > > > Gradle appears to be build system, AFAIK more popular in > Java > > > > > world. > > > > > > > > > meanwhile, Apache Incubator MXNet project uses CMake as i= ts > > > build > > > > > > > system. > > > > > > > > > please correct me, I am wrong. > > > > > > > > > in general, conan, as a package manager, is pretty > > > > > > > build-system-agnostic, > > > > > > > > > and it may work with arbitrary build systems (to count a > few, > > > > > CMake, > > > > > > > > > premake, qmake, full list: > > > > > > > > > https://docs.conan.io/en/latest/reference/generators.html > ). > > I > > > > > don't > > > > > > > > think > > > > > > > > > Gradle is exception here. > > > > > > > > > also, for instance, Android Studio also uses Gradle for > > Android > > > > C++ > > > > > > > > > projects, and conan works flawlessly in this particular > case > > ( > > > > > > > > > > > > > > > > > > > > > > > > > https://blog.conan.io/2018/02/13/Android-Studio-project-Conan-Boost.html > > > > > > > > ). > > > > > > > > > > > > > > > > > > yours sincerely, Konstantin > > > > > > > > > > > > > > > > > > =D0=BF=D0=BD, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. = =D0=B2 16:43, Ivan Serdyuk < > > > > > > > local.tourist.kiev@gmail.com > > > > > > > > >: > > > > > > > > > > > > > > > > > > > Kostantin, and what (overall) option with using Gradle? > > Does > > > > this > > > > > > > your > > > > > > > > > > suggested package manager has been supported by Gradle? > > > > > > > > > > > > > > > > > > > > Ivan > > > > > > > > > > > > > > > > > > > > On Mon, Nov 26, 2018 at 9:43 AM Konstantin Ivlev < > > > > > > > tomskside@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > hello, > > > > > > > > > > > > > > > > > > > > > > this email is related to the following PR and JIRA > > ticket: > > > > > > > > > > > - [MXNET-1229] use OpenBLAS, lapack & OpenCV from con= an > > > > > > > > > > > > > > > > > > > > > > - use conan to manage project dependencies > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > conan is an open-source package > > manager > > > > > for C++ > > > > > > > > > > > projects. it allows to manage project dependencies in > > > > > transparent > > > > > > > and > > > > > > > > > > > declarative manner. > > > > > > > > > > > > > > > > > > > > > > currently, apache incubator-mxnet project uses the > > > following > > > > > > > > different > > > > > > > > > > > ways to manage its dependencies: > > > > > > > > > > > > > > > > > > > > > > - download GitHub archives during the build > > > > > > > > > > > - OpenBLAS > > > > > > > > > > > - OpenCV > > > > > > > > > > > - conda (alternative way to GitHu= b > > > > > archives) > > > > > > > > > > > - download from CMake > > > > > > > > > > > - Intel Math Kernel Library < > > > > > > > > https://software.intel.com/en-us/mkl> > > > > > > > > > > > (MKL) > > > > > > > > > > > - Git submodules > > > > > > > > > > > - cub > > > > > > > > > > > - dlpack > > > > > > > > > > > - dmlc-core > > > > > > > > > > > - googletest > > > > > > > > > > > > - mkldnn > > > > > > > > > > > - mshadow > > > > > > > > > > > - onnx-tensorrt < > > https://github.com/onnx/onnx-tensorrt > > > > > > > > > > > > > > > - openmp > > > > > > > > > > > - ps-lite > > > > > > > > > > > - tvm > > > > > > > > > > > > > > > > > > > > > > this appears to be very heterogeneous and hard to > > > > > manage/maintain, > > > > > > > as > > > > > > > > > > > multiple various commands are in use to achieve > > > dependencies > > > > > > > > > > installation, > > > > > > > > > > > as well as multiple places are to look for dependency > > > > versions > > > > > and > > > > > > > > their > > > > > > > > > > > updates. > > > > > > > > > > > > > > > > > > > > > > with conan, it may became much more straightforward, = as > > > > > > > dependencies > > > > > > > > will > > > > > > > > > > > be declared in single place (conanfile) and installed > via > > > > > single > > > > > > > > command > > > > > > > > > > > (conan install). > > > > > > > > > > > > > > > > > > > > > > as project is very complex, and has lots of > dependencies, > > > for > > > > > the > > > > > > > > first > > > > > > > > > > > prototype I've used only very few of dependencies fro= m > > > conan: > > > > > > > OpenCV > > > > > > > > > > > < > > https://bintray.com/conan-community/conan/opencv%3Aconan > > > >, > > > > > > > OpenBLAS > > > > > > > > > > > < > > > https://bintray.com/conan-community/conan/openblas%3Aconan> > > > > > and > > > > > > > > lapack > > > > > > > > > > > < > > https://bintray.com/conan-community/conan/lapack%3Aconan > > > >. > > > > > > > > > > > others may be easily added then one by one, but they > > first > > > > has > > > > > to > > > > > > > be > > > > > > > > > > > packaged (not all of them are packaged yet, e.g. > > GoogleTest > > > > > > > > > > > < > > > > > https://bintray.com/bincrafters/public-conan/gtest%3Abincrafters> > > > > > > > > is > > > > > > > > > > > available, while MKL is not). > > > > > > > > > > > > > > > > > > > > > > I attach patch which adds an initial conan support as > > proof > > > > of > > > > > > > > concept. > > > > > > > > > > > also, I attach two simple build scripts, which I've > used > > to > > > > > test > > > > > > > (for > > > > > > > > > > > Windows and Linux / Mac OS X). Google Mail blocks .sh > and > > > > .cmd > > > > > > > > > > extensions, > > > > > > > > > > > so you'll need to rename files. > > > > > > > > > > > lemme know if you have any further questions. > > > > > > > > > > > > > > > > > > > > > > yours sincerely, Konstantin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --00000000000006c610057bbb332e--