From dev-return-33368-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Apr 13 16:00:47 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6E3C0180627 for ; Fri, 13 Apr 2018 16:00:46 +0200 (CEST) Received: (qmail 49274 invoked by uid 500); 13 Apr 2018 14:00:45 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 49257 invoked by uid 99); 13 Apr 2018 14:00:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Apr 2018 14:00:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 41E99180543 for ; Fri, 13 Apr 2018 14:00:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.192 X-Spam-Level: *** X-Spam-Status: No, score=3.192 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AZuX36_ag4AO for ; Fri, 13 Apr 2018 14:00:40 +0000 (UTC) Received: from mail-qt0-f177.google.com (mail-qt0-f177.google.com [209.85.216.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 21CBB5F4E4 for ; Fri, 13 Apr 2018 14:00:40 +0000 (UTC) Received: by mail-qt0-f177.google.com with SMTP id s48so8107750qtb.10 for ; Fri, 13 Apr 2018 07:00:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=rWBm/C49uGuLsMTuNYznc5maBvLQrii1UFLKeYxL2i8=; b=E9TCeDSHO4N1mnAQtNLEpuOIKt+Q7sSWy7pxcG+AzH/CRU5XVHvOB+WnxWH/VdwOYc pFF7YkH7ri+7jcn/hFnSOFExtLyNv7F7pJKrofeK/jfmUxeu8MFT1uMHZCmHDSM4Tym7 04ez/zu6LkTvTBMvfNOFIaHjYbiKYX9xqXmOgueP8/eU+uLQ6rAKmzZnp7phZymlGCqv oatBBgZzyLKYpzczuPXbN9I/KpTHGrmZjArs2jL6CaR2ZDKKtAt6NotJY4TM7mAyv9Fu NAuD57zHVuokUx9w+g4QoHBeK9myD9rSHhQPc5Fiz65d1iYWAMeUesEXtimdUWZM6pG+ xO0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=rWBm/C49uGuLsMTuNYznc5maBvLQrii1UFLKeYxL2i8=; b=GKRZodi7mFR7dBnxATKrsn2akckqeUj3TdTOlepE5PL1RStTkXYCwLWToxrKLaQFC1 8ZZjkNWtqd5HPSUlOTLAvuv7ZfWLDbxXp6xrg+i59pypcYpaTM2ebngbtpkvEQPGTr/B DAZ7MpNU6KaQb7gZJlm0ZHJVvpaEa6Nf/n85oZKTqsgEmov8Tet3nPIGmdhcvZya8NpB StM5pI5LxJZvbUw3YLbhg2rR6hWWDZeMkTOXfXURqMx8ZBNQFWB5OBA4NvKeMXXHF7jJ biaeh3l1/a+CGd/7dJyp+UYOfafy6K60d6jua5O0/PsCR1TWo+WLjjD/OOcI19VFJQ4n pR5A== X-Gm-Message-State: ALQs6tDSsYMDK9068od77Oy0SZ2b9DDjfLKrd4E57CZCzuONZ6x9206b 75qymVQy3pqjQycwQIVDQd0s+s5uC1pdAmWdAac= X-Google-Smtp-Source: AIpwx4+Z9q0XIF8N1y/BRWHRDnaCLV3vZq8RdvZhbf2FiV3qdOWGgcmJ/fA+1JYgc39+m5eCeJRHFbw0jurqsuxMyG4= X-Received: by 10.200.52.197 with SMTP id x5mr3205373qtb.322.1523628039482; Fri, 13 Apr 2018 07:00:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.81.76 with HTTP; Fri, 13 Apr 2018 07:00:39 -0700 (PDT) In-Reply-To: References: <1521020161099-0.post@n4.nabble.com> <6A73B8D9-6ADF-4E6F-99BC-A6A6FE4579DC@gmail.com> <43596C64-F54D-4F35-BC67-001E0952126B@gmail.com> <07B5A6DF-B139-48BE-8294-E25F4378222C@gmail.com> <6650CEC4-B5F2-4350-BD7C-FEEDB6B0F71F@gmail.com> <22A24C0D-A192-4995-8D60-BAC4ED3E6057@gmail.com> <4E3E4089-F52E-4612-BA41-A9DD5A94EA45@gmail.com> From: Ilya Kasnacheev Date: Fri, 13 Apr 2018 17:00:39 +0300 Message-ID: Subject: Re: Apache Ignite RPM packaging (Stage II) To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="001a1141dd4210386d0569bb4c22" --001a1141dd4210386d0569bb4c22 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-04-13 7:42 GMT+03:00 Peter Ivanov : > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev > wrote: > > > > > Moreover I did not find a way to start service if default installed JVM > is > > Java 7 :( I understand it's EOL, still this is something that hit me. > > > Apache Ignite >=3D2.4 does not support Java 7 - it is said in documentati= on, > DEVNOTES and even in startup scripts. > > I have Java 8 too, but I could not get service from package to start Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that matter). Is it possible to specify it while running packaged Ignite? > > > > > apache-ignite-libs is a totally unexpected package name. apache-ignite > core > > doesn't depend on it. It doesn't enable anything out of the box. The > > package is huge. > > =E2=80=98apache-ignite-libs=E2=80=99 is an aggregation package (for now) = for all optional > libs we are delivering. Possibly later they will be split more granular o= r > even package per lib (like php, perl, python, etc. do for their libs). > This package dependency on =E2=80=98apache-ignite-core=E2=80=99 may seem = confusing though, > I will try to explain it in IEP at least for current iteration. > Okay, but how do you add optional libs to be included into Ignite classpath while being launched by service? Is it even possible? If it isn't, I think it doesn't make sense to ship apache-ignite-libs at all. > > Further naming may become clear when we=E2=80=99ll start initiative on in= cluding > packages to popular Linux distributions and theirs community will join > naming discussions. > Renaming packages once they're deployed widely will be a pain point to out users. Some things should probably be thought out in advance. > > > > > > > Frankly speaking, I'm not sure that improvements over Stage I are enoug= h > as > > of now. For demo-like activity, we can probably go with one package fit= s > > all. > > > > The process of finding the best package architecture is iterative, but > previously community agreed in split design proposed for 2.5 release. > > Also, split architecture is half of proposed improvements. The other half= - > new process for deploying packages to Bintray (with virtually indefinite > storage capabilities). > I think we could drop the split for now, or at least drop apache-ignite-libs package at all. Probably also drop apache-ignite-cpp package and maybe apache-ignite-benchmarks. The point of a package is to ship something into root file system that can be used from root file system. If cpp files require compilation we should not ship them, or ship them to 'examples'. Ditto with benchmarks. If there's no mechanism to add optional libs to Ignite classpath, we should not ship optional libs. Moreover, some of 'optional' modules such as yarn don't make sense here because they're not supposed to be used with standalone Ignite. IMO it is not right to try and shove every file from Ignite distribution into some package. We should only put in packages things that can be used. If something can't be used without copying it to a different FS location, it should be in examples or not packaged at all. In my opinion, it doesn't make sense to implement an underwhelming package split right now just because we have agreed to have *some* package split in 2.5. Let's aim for happiness. > > > > > > > -- > > Ilya Kasnacheev > > > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov : > > > > > If someone from PMC=D1=8B or Committers still sees necessity about in= cluding > > > these tasks into Apache Ignite 2.5 release, this is the last chance t= o > do > > > so. > > > Otherwise this task will be moved to at 2.6 release at least, or even > > > moved to backlog indefinitely. > > > > > > > > > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov wrote: > > > > > > > > To top new RPM architecture off, update to release process is > > introduced > > > =E2=80=94 [1] [2]. > > > > > > > > Both tasks (this one and IGNITE-7647) are ready for review and shou= ld > > be > > > merged simultaneously. > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172 > > > > [2] https://github.com/apache/ignite-release/pull/1 > > > > > > > > > > > > > > > > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev > > > > wrote: > > > >> > > > >> Hello! > > > >> > > > >> Let me share my idea of how this shoud work. Splitting package int= o > > > >> sub-packages should be dependency-driven. > > > >> > > > >> It means that all Ignite modules without dependencies or with smal= l > > > >> dependencies (such as ignite-log4j) should be included in > ignite-core. > > > It > > > >> doesn't make sense to make a zillion RPM packages. > > > >> > > > >> Critical things like ignite-spring and ignite-indexing should be i= n > > > >> ignite-core of course, even if they have dependencies. Ignite-core > > > should > > > >> be fully self-sufficient and feature-complete. > > > >> > > > >> However, e.g. .net API should probably be in a separate package, > > > because it > > > >> should depend on mono | net-core. We may also have ignite-devel > > package > > > >> which should include all modules which only make sense for > developers > > > who > > > >> write code. Such as hibernate integration. > > > >> > > > >> I'm not sure about MR modules. The main question should be, does i= t > > have > > > >> dependencies? Can it run stand-alone without writing code? > > > >> > > > >> Hope this helps, > > > >> > > > >> -- > > > >> Ilya Kasnacheev > > > >> > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov : > > > >> > > > >>> Hi, Igniters! > > > >>> > > > >>> > > > >>> Here are some news on our RPM packages initiative. > > > >>> > > > >>> 1. I=E2=80=99ve finished preliminary developing of Stage II versi= on of RPM > > > >>> packages [1]. Main =E2=80=9Cnew feature=E2=80=9D is =E2=80=94 spl= it design. Also I=E2=80=99ve added > > > >>> package.sh script for automating package building process which > will > > > help > > > >>> organise corresponding builds in TC as well as simplify process f= or > > > >>> developers who wishes to have custom packages. > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with > > > Apache > > > >>> Ignite 2.5 release, I=E2=80=99d greatly appreciate your help in f= inding > > > reviewer. > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [= 4] > > > >>> repositories on Apache Bintray. Though they are already prepared > for > > > >>> hosting RPM and DEB packages respectively, and there is a way of > > > linking > > > >>> them to apache.org/dist/ignite page, there is possible alternativ= e > > in > > > >>> storing there only plain directory layout corresponding to each > > > repository > > > >>> type (RPM and DEB) and manage this layout (repodata, distribution= s, > > > >>> versions, etc.) by ourselves, having more control over repositori= es > > but > > > >>> lacking some simplicity of deploying new releases. WDYT? Should w= e > > try > > > >>> Cassandra approach? They are storing their DEB packages as I > > described > > > >>> above [5]. > > > >>> > > > >>> Also =E2=80=94 a question arose while I was working on this issue= : which > OSes > > > (and > > > >>> which versions of each) are we going to support (if we are going) > in > > > terms > > > >>> of step-by-step list? Currently RPM packages are tested only with > > > latest > > > >>> CentOS (and, respectively =E2=80=94 RHEL), but there are a lot mo= re > RPM-based > > > >>> distributives [6] some of which are more o less popular among OS > > > community > > > >>> (ALT, Fedora, openSUSE, etc.). > > > >>> > > > >>> > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647 > > > >>> [2] https://github.com/apache/ignite/pull/3703 > > > >>> [3] https://bintray.com/apache/ignite-rpm > > > >>> [4] https://bintray.com/apache/ignite-deb > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/ > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_ > > > distributions > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov > wrote: > > > >>>> > > > >>>> I suppose that most everything if not all from libs/options will > go > > to > > > >>> OPTIONAL (I=E2=80=99d call it simply =E2=80=98apache-ignite-libs'= ). > > > >>>> More precise lib selection (if something from optional would > better > > to > > > >>> have in core package) will be discussed right after preliminary > split > > > >>> architecture agreement. > > > >>>> > > > >>>> > > > >>>> > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov > > > wrote: > > > >>>>> > > > >>>>> I like idea of keeping simple system of modules, so +1 from me. > > > >>>>> > > > >>>>> Where optional libs (e.g Direct IO plugin) would be included, > would > > > it > > > >>> be > > > >>>>> core or optional? > > > >>>>> > > > >>>>> =D1=87=D1=82, 15 =D0=BC=D0=B0=D1=80. 2018 =D0=B3. =D0=B2 22:09,= Denis Magda : > > > >>>>> > > > >>>>>>> > > > >>>>>>>> > > > >>>>>>>> How big would be a final core module? > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and > create > > > it=E2=80=99s > > > >>> own > > > >>>>>>> package. > > > >>>>>> > > > >>>>>> > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with > Petr > > > and > > > >>>>>> propose the simplest modular system: > > > >>>>>> > > > >>>>>> - core module that includes basic Ignite capabilities includin= g > > SQL, > > > >>>>>> compute grid, service grid, k/v > > > >>>>>> - optional module hosts the rest - ML, streamers integration > > (kafka, > > > >>>>>> flink), kubernetes, etc. > > > >>>>>> > > > >>>>>> What do you think? > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Denis > > > >>>>>> > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov < > > mr.weider@gmail.com> > > > >>> wrote: > > > >>>>>> > > > >>>>>>> *DEB package > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov > > > wrote: > > > >>>>>>>> > > > >>>>>>>> Considering that DEV package for now is almost platform > > > independent > > > >>>>>> (its > > > >>>>>>> a java application more or less), that package will work almo= st > > on > > > any > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian, > > etc. > > > >>>>>>>> The only restriction is existence of systemctl (systemd) > service > > > >>>>>> manager > > > >>>>>>> =E2=80=94 we are dependent on it. > > > >>>>>>>> > > > >>>>>>>> Thats why, for instance, our RPM repository is called simply > > =E2=80=98rpm=E2=80=99 > > > >>> and > > > >>>>>>> package has no arch or dist suffix =E2=80=94 it will work on = CentOS, > > RHEL, > > > >>>>>> Fedora, > > > >>>>>>> etc. with presence of aforementioned systemd. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan < > > > dsetrakyan@apache.org> > > > >>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> Will Debian package work for Ubuntu? > > > >>>>>>>>> > > > >>>>>>>>> D. > > > >>>>>>>>> > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov < > > > mr.weider@gmail.com> > > > >>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to > > > official > > > >>>>>>>>>> repositories, there can be a problem from OS community. > > > >>>>>>>>>> > > > >>>>>>>>>> Concerning DEB packages =E2=80=94 I plan to use RPM as bas= e for DEB > > > package > > > >>>>>>> build > > > >>>>>>>>>> (package layout / install scripts) for speeding up things > and > > > >>>>>> excluding > > > >>>>>>>>>> possible duplication and desynchronisation, so its a matte= r > of > > > =E2=80=99sit > > > >>>>>>> and do=E2=80=99 > > > >>>>>>>>>> rather then some technical research. Thats why I rose > > discussion > > > >>>>>> about > > > >>>>>>>>>> future package architecture, so that after agreement I'm b= e > > > able to > > > >>>>>>> pack > > > >>>>>>>>>> both RPM and DEB identically. > > > >>>>>>>>>> > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according to > > > current > > > >>> RPM > > > >>>>>>>>>> layout in no time. > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan < > > > >>> dsetrakyan@apache.org> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Peter, > > > >>>>>>>>>>> > > > >>>>>>>>>>> I don't think the package size of 280M is going to be a > > > problem at > > > >>>>>>> all, > > > >>>>>>>>>> but > > > >>>>>>>>>>> what you are suggesting can be an improvement down the > road. > > > >>>>>>>>>>> > > > >>>>>>>>>>> In the mean time, I think our top priority should be to > > provide > > > >>>>>>> packages > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly > enough. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Agree? > > > >>>>>>>>>>> > > > >>>>>>>>>>> D. > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider < > > mr.weider@gmail.com> > > > >>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Hi, Igniters! > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, > so > > > I'd > > > >>>>>> like > > > >>>>>>> to > > > >>>>>>>>>>>> move > > > >>>>>>>>>>>> forward to further improve and widen AI delivery through > > > >>> packages. > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing > about > > > >>> 280M+ > > > >>>>>>> and, > > > >>>>>>>>>> to > > > >>>>>>>>>>>> improve usability and significantly reduce required > download > > > >>>>>> sizes, I > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delive= ry > > as > > > >>>>>>> follows: > > > >>>>>>>>>>>> - CORE > > > >>>>>>>>>>>> - bin > > > >>>>>>>>>>>> - config > > > >>>>>>>>>>>> - libs (!optional) > > > >>>>>>>>>>>> - OPTIONAL LIBS > > > >>>>>>>>>>>> - BENCHMARKS > > > >>>>>>>>>>>> - DOCS (?) > > > >>>>>>>>>>>> - EXAMPLES > > > >>>>>>>>>>>> - .NET PLATFORM FILES > > > >>>>>>>>>>>> - C++ PLATFORM FILES > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no > > > reason > > > >>> to > > > >>>>>>>>>> download > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core nod= e > > > >>>>>>> functionality) > > > >>>>>>>>>>>> and > > > >>>>>>>>>>>> maintainability (you are in full control of what is > > installed > > > on > > > >>>>>> your > > > >>>>>>>>>>>> system). > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> After successful architecture choice, same scheme are > > planned > > > to > > > >>> be > > > >>>>>>>>>> used in > > > >>>>>>>>>>>> DEB packages as well. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> WDYT? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> -- > > > >>>>>>>>>>>> Sent from: > > http://apache-ignite-developers.2346864.n4.nabble. > > > >>> com/ > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>> > > > >>> > > > > > > > > > > > > > --001a1141dd4210386d0569bb4c22--