ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Ivanov <mr.wei...@gmail.com>
Subject Re: Apache Ignite RPM packaging (Stage II)
Date Sat, 14 Apr 2018 05:03:56 GMT
Current packages design (after installation) does not differ from binary
archive - everything works (except necessity to run service instead
ignite.sh) just the same way, including libs/optional.

Also, there can be issues with system JDK version by default, but every
problem will be in journalctl and/or  /var/log, and package has strong
dependency on any implementation of JDK 1.8.


I am lacking enough feedback about Apache Ignite from packages from real
users, don’t know real use cases so still "moving in the dark".


On Fri, 13 Apr 2018 at 22:18, Denis Magda <dmagda@apache.org> wrote:

> Ilya,
>
> Thanks for your inputs. The reason why we decided to split Ignite into
> several packages mimics the reason why Java community introduced modular
> subsystem for JDK. That's all about size. Ignite distribution is too big,
> and we're trying to separate it into several components so that people can
> install only the features they need.
>
> The point of a package is to ship something into root file system that can
> > be used from root file system. If cpp files require compilation we should
> > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > there's no mechanism to add optional libs to Ignite classpath, we should
> > not ship optional libs. Moreover, some of 'optional' modules such as yarn
> > don't make sense here because they're not supposed to be used with
> > standalone Ignite.
>
>
> Agree that we need to ship the code that is ready to be run. As for the
> classpath thing, if an optional package is installed into the root (core)
> package directory, then its jars have to be added to "ignite/libs" folder.
> After that, the one needs to restart a cluster node, nd it will add the
> just installed optional libs to the classpath. *Petr*, does it work this
> way or can be implemented this way to address Ilya's concerns?
>
> --
> Denis
>
> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> wrote:
>
> > 2018-04-13 7:42 GMT+03:00 Peter Ivanov <mr.weider@gmail.com>:
> >
> > > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com
> > >
> > > wrote:
> > >
> > > >
> > > > Moreover I did not find a way to start service if default installed
> JVM
> > > is
> > > > Java 7 :( I understand it's EOL, still this is something that hit me.
> > >
> > >
> > > Apache Ignite >=2.4 does not support Java 7 - it is said in
> > documentation,
> > > DEVNOTES and even in startup scripts.
> > >
> > >  I have Java 8 too, but I could not get service from package to start
> > Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> > matter). Is it possible to specify it while running packaged Ignite?
> >
> >
> > >
> > > >
> > > > apache-ignite-libs is a totally unexpected package name.
> apache-ignite
> > > core
> > > > doesn't depend on it. It doesn't enable anything out of the box. The
> > > > package is huge.
> > >
> > > ‘apache-ignite-libs’ is an aggregation package (for now) for all
> optional
> > > libs we are delivering. Possibly later they will be split more granular
> > or
> > > even package per lib (like php, perl, python, etc. do for their libs).
> > > This package dependency on ‘apache-ignite-core’ may seem confusing
> > though,
> > > I will try to explain it in IEP at least for current iteration.
> > >
> >
> > Okay, but how do you add optional libs to be included into Ignite
> classpath
> > while being launched by service? Is it even possible? If it isn't, I
> think
> > it doesn't make sense to ship apache-ignite-libs at all.
> >
> >
> > >
> > > Further naming may become clear when we’ll start initiative on
> including
> > > packages to popular Linux distributions and theirs community will join
> > > naming discussions.
> > >
> > Renaming packages once they're deployed widely will be a pain point to
> out
> > users. Some things should probably be thought out in advance.
> >
> >
> > >
> > >
> > >
> > > >
> > > > Frankly speaking, I'm not sure that improvements over Stage I are
> > enough
> > > as
> > > > of now. For demo-like activity, we can probably go with one package
> > fits
> > > > all.
> > > >
> > >
> > > The process of finding the best package architecture is iterative, but
> > > previously community agreed in split design proposed for 2.5 release.
> > >
> > > Also, split architecture is half of proposed improvements. The other
> > half -
> > > new process for deploying packages to Bintray (with virtually
> indefinite
> > > storage capabilities).
> > >
> > I think we could drop the split for now, or at least drop
> > apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
> > package and maybe apache-ignite-benchmarks.
> >
> > The point of a package is to ship something into root file system that
> can
> > be used from root file system. If cpp files require compilation we should
> > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > there's no mechanism to add optional libs to Ignite classpath, we should
> > not ship optional libs. Moreover, some of 'optional' modules such as yarn
> > don't make sense here because they're not supposed to be used with
> > standalone Ignite.
> >
> > IMO it is not right to try and shove every file from Ignite distribution
> > into some package. We should only put in packages things that can be
> used.
> > If something can't be used without copying it to a different FS location,
> > it should be in examples or not packaged at all.
> >
> > In my opinion, it doesn't make sense to implement an underwhelming
> package
> > split right now just because we have agreed to have *some* package split
> in
> > 2.5. Let's aim for happiness.
> >
> >
> > >
> > >
> > >
> > > >
> > > > --
> > > > Ilya Kasnacheev
> > >
> >
> >
> >
> > > >
> > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <mr.weider@gmail.com>:
> > > >
> > > > > If someone from PMCы or Committers still sees necessity about
> > including
> > > > > these tasks into Apache Ignite 2.5 release, this is the last chance
> > to
> > > do
> > > > > so.
> > > > > Otherwise this task will be moved to at 2.6 release at least, or
> even
> > > > > moved to backlog indefinitely.
> > > > >
> > > > >
> > > > >
> > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <mr.weider@gmail.com>
> wrote:
> > > > > >
> > > > > > To top new RPM architecture off, update to release process is
> > > > introduced
> > > > > — [1] [2].
> > > > > >
> > > > > > Both tasks (this one and IGNITE-7647) are ready for review and
> > should
> > > > be
> > > > > merged simultaneously.
> > > > > >
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > > > [2] https://github.com/apache/ignite-release/pull/1
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> > ilya.kasnacheev@gmail.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> Hello!
> > > > > >>
> > > > > >> Let me share my idea of how this shoud work. Splitting package
> > into
> > > > > >> sub-packages should be dependency-driven.
> > > > > >>
> > > > > >> It means that all Ignite modules without dependencies or
with
> > small
> > > > > >> dependencies (such as ignite-log4j) should be included in
> > > ignite-core.
> > > > > It
> > > > > >> doesn't make sense to make a zillion RPM packages.
> > > > > >>
> > > > > >> Critical things like ignite-spring and ignite-indexing should
be
> > in
> > > > > >> ignite-core of course, even if they have dependencies.
> Ignite-core
> > > > > should
> > > > > >> be fully self-sufficient and feature-complete.
> > > > > >>
> > > > > >> However, e.g. .net API should probably be in a separate
package,
> > > > > because it
> > > > > >> should depend on mono | net-core. We may also have ignite-devel
> > > > package
> > > > > >> which should include all modules which only make sense for
> > > developers
> > > > > who
> > > > > >> write code. Such as hibernate integration.
> > > > > >>
> > > > > >> I'm not sure about MR modules. The main question should
be, does
> > it
> > > > have
> > > > > >> dependencies? Can it run stand-alone without writing code?
> > > > > >>
> > > > > >> Hope this helps,
> > > > > >>
> > > > > >> --
> > > > > >> Ilya Kasnacheev
> > > > > >>
> > > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <mr.weider@gmail.com>:
> > > > > >>
> > > > > >>> Hi, Igniters!
> > > > > >>>
> > > > > >>>
> > > > > >>> Here are some news on our RPM packages initiative.
> > > > > >>>
> > > > > >>> 1. I’ve finished preliminary developing of Stage II
version of
> > RPM
> > > > > >>> packages [1]. Main “new feature” is — split design.
Also I’ve
> > added
> > > > > >>> package.sh script for automating package building process
which
> > > will
> > > > > help
> > > > > >>> organise corresponding builds in TC as well as simplify
process
> > for
> > > > > >>> developers who wishes to have custom packages.
> > > > > >>> PR#3703 [2] is ready for review. Denis, in order to
catch up
> with
> > > > > Apache
> > > > > >>> Ignite 2.5 release, I’d greatly appreciate your help
in finding
> > > > > reviewer.
> > > > > >>> 2. With the help of ASF INFRA team, we now have RPM
[3] and DEB
> > [4]
> > > > > >>> repositories on Apache Bintray. Though they are already
> prepared
> > > for
> > > > > >>> hosting RPM and DEB packages respectively, and there
is a way
> of
> > > > > linking
> > > > > >>> them to apache.org/dist/ignite page, there is possible
> > alternative
> > > > in
> > > > > >>> storing there only plain directory layout corresponding
to each
> > > > > repository
> > > > > >>> type (RPM and DEB) and manage this layout (repodata,
> > distributions,
> > > > > >>> versions, etc.) by ourselves, having more control over
> > repositories
> > > > but
> > > > > >>> lacking some simplicity of deploying new releases. WDYT?
Should
> > we
> > > > try
> > > > > >>> Cassandra approach? They are storing their DEB packages
as I
> > > > described
> > > > > >>> above [5].
> > > > > >>>
> > > > > >>> Also — a question arose while I was working on this
issue:
> which
> > > OSes
> > > > > (and
> > > > > >>> which versions of each) are we going to support (if
we are
> going)
> > > in
> > > > > terms
> > > > > >>> of step-by-step list? Currently RPM packages are tested
only
> with
> > > > > latest
> > > > > >>> CentOS (and, respectively — RHEL), but there are a
lot more
> > > RPM-based
> > > > > >>> distributives [6] some of which are more o less popular
among
> OS
> > > > > community
> > > > > >>> (ALT, Fedora, openSUSE, etc.).
> > > > > >>>
> > > > > >>>
> > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > > > >>> [4] https://bintray.com/apache/ignite-deb
> > > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > > > distributions
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <mr.weider@gmail.com>
> > > wrote:
> > > > > >>>>
> > > > > >>>> I suppose that most everything if not all from libs/options
> will
> > > go
> > > > to
> > > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > > > >>>> More precise lib selection (if something from optional
would
> > > better
> > > > to
> > > > > >>> have in core package) will be discussed right after
preliminary
> > > split
> > > > > >>> architecture agreement.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> dpavlov.spb@gmail.com
> > >
> > > > > wrote:
> > > > > >>>>>
> > > > > >>>>> I like idea of keeping simple system of modules,
so +1 from
> me.
> > > > > >>>>>
> > > > > >>>>> Where optional libs (e.g Direct IO plugin) would
be included,
> > > would
> > > > > it
> > > > > >>> be
> > > > > >>>>> core or optional?
> > > > > >>>>>
> > > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda
<dmagda@apache.org
> >:
> > > > > >>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> How big would be a final core module?
> > > > > >>>>>>> Around 30M. Can be shrinked to ~15M
if separate Visor and
> > > create
> > > > > it’s
> > > > > >>> own
> > > > > >>>>>>> package.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference.
 I would agree
> with
> > > Petr
> > > > > and
> > > > > >>>>>> propose the simplest modular system:
> > > > > >>>>>>
> > > > > >>>>>> - core module that includes basic Ignite
capabilities
> > including
> > > > SQL,
> > > > > >>>>>> compute grid, service grid, k/v
> > > > > >>>>>> - optional module hosts the rest - ML, streamers
integration
> > > > (kafka,
> > > > > >>>>>> flink), kubernetes, etc.
> > > > > >>>>>>
> > > > > >>>>>> What do you think?
> > > > > >>>>>>
> > > > > >>>>>> --
> > > > > >>>>>> Denis
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov
<
> > > > mr.weider@gmail.com>
> > > > > >>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> *DEB package
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov
<
> mr.weider@gmail.com>
> > > > > wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Considering that DEV package for
now is almost platform
> > > > > independent
> > > > > >>>>>> (its
> > > > > >>>>>>> a java application more or less), that
package will work
> > almost
> > > > on
> > > > > any
> > > > > >>>>>>> DEB-based linux, including but not limited
to Ubuntu,
> Debian,
> > > > etc.
> > > > > >>>>>>>> The only restriction is existence
of systemctl (systemd)
> > > service
> > > > > >>>>>> manager
> > > > > >>>>>>> — we are dependent on it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thats why, for instance, our RPM
repository is called
> simply
> > > > ‘rpm’
> > > > > >>> and
> > > > > >>>>>>> package has no arch or dist suffix —
it will work on
> CentOS,
> > > > RHEL,
> > > > > >>>>>> Fedora,
> > > > > >>>>>>> etc. with presence of aforementioned
systemd.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy
Setrakyan <
> > > > > dsetrakyan@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Will Debian package work for
Ubuntu?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> D.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52
PM, Petr Ivanov <
> > > > > mr.weider@gmail.com>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Not a problem, rather nuisance.
Also, when we will move
> to
> > > > > official
> > > > > >>>>>>>>>> repositories, there can
be a problem from OS community.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Concerning DEB packages
— I plan to use RPM as base for
> > DEB
> > > > > package
> > > > > >>>>>>> build
> > > > > >>>>>>>>>> (package layout / install
scripts) for speeding up
> things
> > > and
> > > > > >>>>>> excluding
> > > > > >>>>>>>>>> possible duplication and
desynchronisation, so its a
> > matter
> > > of
> > > > > ’sit
> > > > > >>>>>>> and do’
> > > > > >>>>>>>>>> rather then some technical
research. Thats why I rose
> > > > discussion
> > > > > >>>>>> about
> > > > > >>>>>>>>>> future package architecture,
so that after agreement I'm
> > be
> > > > > able to
> > > > > >>>>>>> pack
> > > > > >>>>>>>>>> both RPM and DEB identically.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Yet, if you insist, I can
create DEB package according
> to
> > > > > current
> > > > > >>> RPM
> > > > > >>>>>>>>>> layout in no time.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53,
Dmitriy Setrakyan <
> > > > > >>> dsetrakyan@apache.org>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Peter,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I don't think the package
size of 280M is going to be a
> > > > > problem at
> > > > > >>>>>>> all,
> > > > > >>>>>>>>>> but
> > > > > >>>>>>>>>>> what you are suggesting
can be an improvement down the
> > > road.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> In the mean time, I
think our top priority should be to
> > > > provide
> > > > > >>>>>>> packages
> > > > > >>>>>>>>>>> for Debian and Ubuntu.
Having only RPMs is not nearly
> > > enough.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Agree?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> D.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Wed, Mar 14, 2018
at 5:36 AM, vveider <
> > > > mr.weider@gmail.com>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hi, Igniters!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Release 2.4 is almost
there, at least binary part of
> it,
> > > so
> > > > > I'd
> > > > > >>>>>> like
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>>> move
> > > > > >>>>>>>>>>>> forward to further
improve and widen AI delivery
> through
> > > > > >>> packages.
> > > > > >>>>>>>>>>>> As of now, Apache
Ignite ships in RPM package weighing
> > > about
> > > > > >>> 280M+
> > > > > >>>>>>> and,
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>>> improve usability
and significantly reduce required
> > > download
> > > > > >>>>>> sizes, I
> > > > > >>>>>>>>>>>> purpose that in
2.5 release we introduce splitted
> > delivery
> > > > as
> > > > > >>>>>>> follows:
> > > > > >>>>>>>>>>>> - CORE
> > > > > >>>>>>>>>>>> - bin
> > > > > >>>>>>>>>>>> - config
> > > > > >>>>>>>>>>>> - libs (!optional)
> > > > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > > > >>>>>>>>>>>> - BENCHMARKS
> > > > > >>>>>>>>>>>> - DOCS (?)
> > > > > >>>>>>>>>>>> - EXAMPLES
> > > > > >>>>>>>>>>>> - .NET PLATFORM
FILES
> > > > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> This architecture,
as I assume, will add flexibility
> (no
> > > > > reason
> > > > > >>> to
> > > > > >>>>>>>>>> download
> > > > > >>>>>>>>>>>> all 280M+ of binaries
where you are to run only core
> > node
> > > > > >>>>>>> functionality)
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>> maintainability
(you are in full control of what is
> > > > installed
> > > > > on
> > > > > >>>>>> your
> > > > > >>>>>>>>>>>> system).
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> After successful
architecture choice, same scheme are
> > > > planned
> > > > > to
> > > > > >>> be
> > > > > >>>>>>>>>> used in
> > > > > >>>>>>>>>>>> DEB packages as
well.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> --
> > > > > >>>>>>>>>>>> Sent from:
> > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > >>> com/
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message