ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vyacheslav Daradur <daradu...@gmail.com>
Subject Re: Data compression in Ignite 2.0
Date Mon, 15 May 2017 16:18:48 GMT
Dmitriy,

I have ready prototype. I want to show it.
It is always easier to discuss on example.

2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:

> Vyacheslav,
>
> I think it is a bit premature to provide a PR without getting a community
> consensus on the dev list. Please allow some time for the community to
> respond.
>
> D.
>
> On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <daradurvs@gmail.com>
> wrote:
>
> > I created the ticket: https://issues.apache.org/jira/browse/IGNITE-5226
> >
> > I'll prepare a PR with described solution in couple of days.
> >
> > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
> >
> > > Hi, Igniters!
> > >
> > > Apache 2.0 is released.
> > >
> > > Let's continue the discussion about a compression design.
> > >
> > > At the moment, I found only one solution which is compatible with
> > querying
> > > and indexing, this is per-objects-field compression.
> > > Per-fields compression means that metadata (a header) of an object
> won't
> > > be compressed, only serialized values of an object fields (in bytes
> array
> > > form) will be compressed.
> > >
> > > This solution have some contentious issues:
> > > - small values, like primitives and short arrays - there isn't sense to
> > > compress them;
> > > - there is no possible to use compression with java-predefined types;
> > >
> > > We can provide an annotation, @IgniteCompression - for example, which
> can
> > > be used by users for marking fields to compress.
> > >
> > > Any thoughts?
> > >
> > > Maybe someone already have ready design?
> > >
> > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
> > >
> > >> Alexey,
> > >>
> > >> Yes, I've read it.
> > >>
> > >> Ok, let's discuss about public API design.
> > >>
> > >> I think we need to add some a configure entity to CacheConfiguration,
> > >> which will contain the Compressor interface implementation and some
> > usefull
> > >> parameters.
> > >> Or maybe to provide a BinaryMarshaller decorator, which will be
> compress
> > >> data after marshalling.
> > >>
> > >>
> > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov <akuznetsov@apache.org>:
> > >>
> > >>> Vyacheslav,
> > >>>
> > >>> Did you read initial discussion [1] about compression?
> > >>> As far as I remember we agreed to add only some "top-level" API in
> > order
> > >>> to
> > >>> provide a way for
> > >>> Ignite users to inject some sort of custom compression.
> > >>>
> > >>>
> > >>> [1]
> > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c
> > >>> ompression-in-Ignite-2-0-td10099.html
> > >>>
> > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs <daradurvs@gmail.com>
> > wrote:
> > >>>
> > >>> > Hi Igniters!
> > >>> >
> > >>> > I am interested in this task.
> > >>> > Provide some kind of pluggable compression SPI support
> > >>> > <https://issues.apache.org/jira/browse/IGNITE-3592>
> > >>> >
> > >>> > I developed a solution on BinaryMarshaller-level, but reviewer
has
> > >>> rejected
> > >>> > it.
> > >>> >
> > >>> > Let's continue discussion of task goals and solution design.
> > >>> > As I understood that, the main goal of this task is to store data
> in
> > >>> > compressed form.
> > >>> > This is what I need from Ignite as its user. Compression provides
> > >>> economy
> > >>> > on
> > >>> > servers.
> > >>> > We can store more data on same servers at the cost of increasing
> CPU
> > >>> > utilization.
> > >>> >
> > >>> > I'm researching a possibility of implementation of compression
at
> the
> > >>> > cache-level.
> > >>> >
> > >>> > Any thoughts?
> > >>> >
> > >>> > --
> > >>> > Best regards,
> > >>> > Vyacheslav
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > View this message in context: http://apache-ignite-
> > >>> > developers.2346864.n4.nabble.com/Data-compression-in-
> > >>> > Ignite-2-0-tp10099p16317.html
> > >>> > Sent from the Apache Ignite Developers mailing list archive at
> > >>> Nabble.com.
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Alexey Kuznetsov
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards, Vyacheslav
> > >>
> > >
> > >
> > >
> > > --
> > > Best Regards, Vyacheslav
> > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav
> >
>



-- 
Best Regards, Vyacheslav

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message