ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Apache Ignite RPM packaging (Stage II)
Date Fri, 13 Apr 2018 14:00:39 GMT
2018-04-13 7:42 GMT+03:00 Peter Ivanov <mr.weider@gmail.com>:

> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
> wrote:
>
> >
> > Moreover I did not find a way to start service if default installed JVM
> is
> > Java 7 :( I understand it's EOL, still this is something that hit me.
>
>
> Apache Ignite >=2.4 does not support Java 7 - it is said in documentation,
> DEVNOTES and even in startup scripts.
>
>  I have Java 8 too, but I could not get service from package to start
Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
matter). Is it possible to specify it while running packaged Ignite?


>
> >
> > apache-ignite-libs is a totally unexpected package name. apache-ignite
> core
> > doesn't depend on it. It doesn't enable anything out of the box. The
> > package is huge.
>
> ‘apache-ignite-libs’ is an aggregation package (for now) for all optional
> libs we are delivering. Possibly later they will be split more granular or
> even package per lib (like php, perl, python, etc. do for their libs).
> This package dependency on ‘apache-ignite-core’ may seem confusing though,
> I will try to explain it in IEP at least for current iteration.
>

Okay, but how do you add optional libs to be included into Ignite classpath
while being launched by service? Is it even possible? If it isn't, I think
it doesn't make sense to ship apache-ignite-libs at all.


>
> Further naming may become clear when we’ll start initiative on including
> packages to popular Linux distributions and theirs community will join
> naming discussions.
>
Renaming packages once they're deployed widely will be a pain point to out
users. Some things should probably be thought out in advance.


>
>
>
> >
> > Frankly speaking, I'm not sure that improvements over Stage I are enough
> as
> > of now. For demo-like activity, we can probably go with one package fits
> > all.
> >
>
> The process of finding the best package architecture is iterative, but
> previously community agreed in split design proposed for 2.5 release.
>
> Also, split architecture is half of proposed improvements. The other half -
> new process for deploying packages to Bintray (with virtually indefinite
> storage capabilities).
>
I think we could drop the split for now, or at least drop
apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
package and maybe apache-ignite-benchmarks.

The point of a package is to ship something into root file system that can
be used from root file system. If cpp files require compilation we should
not ship them, or ship them to 'examples'. Ditto with benchmarks. If
there's no mechanism to add optional libs to Ignite classpath, we should
not ship optional libs. Moreover, some of 'optional' modules such as yarn
don't make sense here because they're not supposed to be used with
standalone Ignite.

IMO it is not right to try and shove every file from Ignite distribution
into some package. We should only put in packages things that can be used.
If something can't be used without copying it to a different FS location,
it should be in examples or not packaged at all.

In my opinion, it doesn't make sense to implement an underwhelming package
split right now just because we have agreed to have *some* package split in
2.5. Let's aim for happiness.


>
>
>
> >
> > --
> > Ilya Kasnacheev
>



> >
> > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <mr.weider@gmail.com>:
> >
> > > If someone from PMCы or Committers still sees necessity about including
> > > these tasks into Apache Ignite 2.5 release, this is the last chance to
> do
> > > so.
> > > Otherwise this task will be moved to at 2.6 release at least, or even
> > > moved to backlog indefinitely.
> > >
> > >
> > >
> > > > On 9 Apr 2018, at 19:08, Petr Ivanov <mr.weider@gmail.com> wrote:
> > > >
> > > > To top new RPM architecture off, update to release process is
> > introduced
> > > — [1] [2].
> > > >
> > > > Both tasks (this one and IGNITE-7647) are ready for review and should
> > be
> > > merged simultaneously.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > [2] https://github.com/apache/ignite-release/pull/1
> > > >
> > > >
> > > >
> > > >
> > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <ilya.kasnacheev@gmail.com
> >
> > > wrote:
> > > >>
> > > >> Hello!
> > > >>
> > > >> Let me share my idea of how this shoud work. Splitting package into
> > > >> sub-packages should be dependency-driven.
> > > >>
> > > >> It means that all Ignite modules without dependencies or with small
> > > >> dependencies (such as ignite-log4j) should be included in
> ignite-core.
> > > It
> > > >> doesn't make sense to make a zillion RPM packages.
> > > >>
> > > >> Critical things like ignite-spring and ignite-indexing should be in
> > > >> ignite-core of course, even if they have dependencies. Ignite-core
> > > should
> > > >> be fully self-sufficient and feature-complete.
> > > >>
> > > >> However, e.g. .net API should probably be in a separate package,
> > > because it
> > > >> should depend on mono | net-core. We may also have ignite-devel
> > package
> > > >> which should include all modules which only make sense for
> developers
> > > who
> > > >> write code. Such as hibernate integration.
> > > >>
> > > >> I'm not sure about MR modules. The main question should be, does it
> > have
> > > >> dependencies? Can it run stand-alone without writing code?
> > > >>
> > > >> Hope this helps,
> > > >>
> > > >> --
> > > >> Ilya Kasnacheev
> > > >>
> > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <mr.weider@gmail.com>:
> > > >>
> > > >>> Hi, Igniters!
> > > >>>
> > > >>>
> > > >>> Here are some news on our RPM packages initiative.
> > > >>>
> > > >>> 1. I’ve finished preliminary developing of Stage II version
of RPM
> > > >>> packages [1]. Main “new feature” is — split design. Also
I’ve added
> > > >>> package.sh script for automating package building process which
> will
> > > help
> > > >>> organise corresponding builds in TC as well as simplify process
for
> > > >>> developers who wishes to have custom packages.
> > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> > > Apache
> > > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > > reviewer.
> > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB
[4]
> > > >>> repositories on Apache Bintray. Though they are already prepared
> for
> > > >>> hosting RPM and DEB packages respectively, and there is a way
of
> > > linking
> > > >>> them to apache.org/dist/ignite page, there is possible alternative
> > in
> > > >>> storing there only plain directory layout corresponding to each
> > > repository
> > > >>> type (RPM and DEB) and manage this layout (repodata, distributions,
> > > >>> versions, etc.) by ourselves, having more control over repositories
> > but
> > > >>> lacking some simplicity of deploying new releases. WDYT? Should
we
> > try
> > > >>> Cassandra approach? They are storing their DEB packages as I
> > described
> > > >>> above [5].
> > > >>>
> > > >>> Also — a question arose while I was working on this issue: which
> OSes
> > > (and
> > > >>> which versions of each) are we going to support (if we are going)
> in
> > > terms
> > > >>> of step-by-step list? Currently RPM packages are tested only with
> > > latest
> > > >>> CentOS (and, respectively — RHEL), but there are a lot more
> RPM-based
> > > >>> distributives [6] some of which are more o less popular among
OS
> > > community
> > > >>> (ALT, Fedora, openSUSE, etc.).
> > > >>>
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > >>> [4] https://bintray.com/apache/ignite-deb
> > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > distributions
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <mr.weider@gmail.com>
> wrote:
> > > >>>>
> > > >>>> I suppose that most everything if not all from libs/options
will
> go
> > to
> > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > >>>> More precise lib selection (if something from optional would
> better
> > to
> > > >>> have in core package) will be discussed right after preliminary
> split
> > > >>> architecture agreement.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <dpavlov.spb@gmail.com>
> > > wrote:
> > > >>>>>
> > > >>>>> I like idea of keeping simple system of modules, so +1
from me.
> > > >>>>>
> > > >>>>> Where optional libs (e.g Direct IO plugin) would be included,
> would
> > > it
> > > >>> be
> > > >>>>> core or optional?
> > > >>>>>
> > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <dmagda@apache.org>:
> > > >>>>>
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> How big would be a final core module?
> > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate
Visor and
> create
> > > it’s
> > > >>> own
> > > >>>>>>> package.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Guys, 30 vs 280M is a huuuuge difference.  I would
agree with
> Petr
> > > and
> > > >>>>>> propose the simplest modular system:
> > > >>>>>>
> > > >>>>>> - core module that includes basic Ignite capabilities
including
> > SQL,
> > > >>>>>> compute grid, service grid, k/v
> > > >>>>>> - optional module hosts the rest - ML, streamers integration
> > (kafka,
> > > >>>>>> flink), kubernetes, etc.
> > > >>>>>>
> > > >>>>>> What do you think?
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Denis
> > > >>>>>>
> > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > mr.weider@gmail.com>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>>> *DEB package
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <mr.weider@gmail.com>
> > > wrote:
> > > >>>>>>>>
> > > >>>>>>>> Considering that DEV package for now is almost
platform
> > > independent
> > > >>>>>> (its
> > > >>>>>>> a java application more or less), that package
will work almost
> > on
> > > any
> > > >>>>>>> DEB-based linux, including but not limited to
Ubuntu, Debian,
> > etc.
> > > >>>>>>>> The only restriction is existence of systemctl
(systemd)
> service
> > > >>>>>> manager
> > > >>>>>>> — we are dependent on it.
> > > >>>>>>>>
> > > >>>>>>>> Thats why, for instance, our RPM repository
is called simply
> > ‘rpm’
> > > >>> and
> > > >>>>>>> package has no arch or dist suffix — it will
work on CentOS,
> > RHEL,
> > > >>>>>> Fedora,
> > > >>>>>>> etc. with presence of aforementioned systemd.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan
<
> > > dsetrakyan@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Will Debian package work for Ubuntu?
> > > >>>>>>>>>
> > > >>>>>>>>> D.
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr
Ivanov <
> > > mr.weider@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Not a problem, rather nuisance. Also,
when we will move to
> > > official
> > > >>>>>>>>>> repositories, there can be a problem
from OS community.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Concerning DEB packages — I plan
to use RPM as base for DEB
> > > package
> > > >>>>>>> build
> > > >>>>>>>>>> (package layout / install scripts)
for speeding up things
> and
> > > >>>>>> excluding
> > > >>>>>>>>>> possible duplication and desynchronisation,
so its a matter
> of
> > > ’sit
> > > >>>>>>> and do’
> > > >>>>>>>>>> rather then some technical research.
Thats why I rose
> > discussion
> > > >>>>>> about
> > > >>>>>>>>>> future package architecture, so that
after agreement I'm be
> > > able to
> > > >>>>>>> pack
> > > >>>>>>>>>> both RPM and DEB identically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yet, if you insist, I can create DEB
package according to
> > > current
> > > >>> RPM
> > > >>>>>>>>>> layout in no time.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy
Setrakyan <
> > > >>> dsetrakyan@apache.org>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Peter,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I don't think the package size
of 280M is going to be a
> > > problem at
> > > >>>>>>> all,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>> what you are suggesting can be
an improvement down the
> road.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> In the mean time, I think our
top priority should be to
> > provide
> > > >>>>>>> packages
> > > >>>>>>>>>>> for Debian and Ubuntu. Having
only RPMs is not nearly
> enough.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Agree?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> D.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM,
vveider <
> > mr.weider@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi, Igniters!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Release 2.4 is almost there,
at least binary part of it,
> so
> > > I'd
> > > >>>>>> like
> > > >>>>>>> to
> > > >>>>>>>>>>>> move
> > > >>>>>>>>>>>> forward to further improve
and widen AI delivery through
> > > >>> packages.
> > > >>>>>>>>>>>> As of now, Apache Ignite ships
in RPM package weighing
> about
> > > >>> 280M+
> > > >>>>>>> and,
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> improve usability and significantly
reduce required
> download
> > > >>>>>> sizes, I
> > > >>>>>>>>>>>> purpose that in 2.5 release
we introduce splitted delivery
> > as
> > > >>>>>>> follows:
> > > >>>>>>>>>>>> - CORE
> > > >>>>>>>>>>>> - bin
> > > >>>>>>>>>>>> - config
> > > >>>>>>>>>>>> - libs (!optional)
> > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > >>>>>>>>>>>> - BENCHMARKS
> > > >>>>>>>>>>>> - DOCS (?)
> > > >>>>>>>>>>>> - EXAMPLES
> > > >>>>>>>>>>>> - .NET PLATFORM FILES
> > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> This architecture, as I assume,
will add flexibility (no
> > > reason
> > > >>> to
> > > >>>>>>>>>> download
> > > >>>>>>>>>>>> all 280M+ of binaries where
you are to run only core node
> > > >>>>>>> functionality)
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>> maintainability (you are in
full control of what is
> > installed
> > > on
> > > >>>>>> your
> > > >>>>>>>>>>>> system).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> After successful architecture
choice, same scheme are
> > planned
> > > to
> > > >>> be
> > > >>>>>>>>>> used in
> > > >>>>>>>>>>>> DEB packages as well.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>> Sent from:
> > http://apache-ignite-developers.2346864.n4.nabble.
> > > >>> com/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message