ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vyacheslav Daradur <daradu...@gmail.com>
Subject Re: Data compression in Ignite 2.0
Date Wed, 24 May 2017 06:49:09 GMT
Guys, any thoughts?

2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:

> Hi guys,
>
> I've prepared the PR to show my idea.
> https://github.com/apache/ignite/pull/1951/files
>
> About querying - I've just copied existing tests and have annotated the
> testing data.
> https://github.com/apache/ignite/pull/1951/files#diff-
> c19a9df4058141d059bb577e75244764
>
> It means fields which will be marked by @BinaryCompression will be
> compressed at marshalling via BinaryMarshaller.
>
> This solution has no effect on existing data or project architecture.
>
> I'll be glad to see your thougths.
>
>
> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>
>> Dmitriy,
>>
>> I have ready prototype. I want to show it.
>> It is always easier to discuss on example.
>>
>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
>>
>>> Vyacheslav,
>>>
>>> I think it is a bit premature to provide a PR without getting a community
>>> consensus on the dev list. Please allow some time for the community to
>>> respond.
>>>
>>> D.
>>>
>>> On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <daradurvs@gmail.com
>>> >
>>> wrote:
>>>
>>> > I created the ticket: https://issues.apache.org/jira
>>> /browse/IGNITE-5226
>>> >
>>> > I'll prepare a PR with described solution in couple of days.
>>> >
>>> > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>>> >
>>> > > Hi, Igniters!
>>> > >
>>> > > Apache 2.0 is released.
>>> > >
>>> > > Let's continue the discussion about a compression design.
>>> > >
>>> > > At the moment, I found only one solution which is compatible with
>>> > querying
>>> > > and indexing, this is per-objects-field compression.
>>> > > Per-fields compression means that metadata (a header) of an object
>>> won't
>>> > > be compressed, only serialized values of an object fields (in bytes
>>> array
>>> > > form) will be compressed.
>>> > >
>>> > > This solution have some contentious issues:
>>> > > - small values, like primitives and short arrays - there isn't sense
>>> to
>>> > > compress them;
>>> > > - there is no possible to use compression with java-predefined types;
>>> > >
>>> > > We can provide an annotation, @IgniteCompression - for example,
>>> which can
>>> > > be used by users for marking fields to compress.
>>> > >
>>> > > Any thoughts?
>>> > >
>>> > > Maybe someone already have ready design?
>>> > >
>>> > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>>> > >
>>> > >> Alexey,
>>> > >>
>>> > >> Yes, I've read it.
>>> > >>
>>> > >> Ok, let's discuss about public API design.
>>> > >>
>>> > >> I think we need to add some a configure entity to
>>> CacheConfiguration,
>>> > >> which will contain the Compressor interface implementation and
some
>>> > usefull
>>> > >> parameters.
>>> > >> Or maybe to provide a BinaryMarshaller decorator, which will be
>>> compress
>>> > >> data after marshalling.
>>> > >>
>>> > >>
>>> > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov <akuznetsov@apache.org
>>> >:
>>> > >>
>>> > >>> Vyacheslav,
>>> > >>>
>>> > >>> Did you read initial discussion [1] about compression?
>>> > >>> As far as I remember we agreed to add only some "top-level"
API in
>>> > order
>>> > >>> to
>>> > >>> provide a way for
>>> > >>> Ignite users to inject some sort of custom compression.
>>> > >>>
>>> > >>>
>>> > >>> [1]
>>> > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c
>>> > >>> ompression-in-Ignite-2-0-td10099.html
>>> > >>>
>>> > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs <daradurvs@gmail.com>
>>> > wrote:
>>> > >>>
>>> > >>> > Hi Igniters!
>>> > >>> >
>>> > >>> > I am interested in this task.
>>> > >>> > Provide some kind of pluggable compression SPI support
>>> > >>> > <https://issues.apache.org/jira/browse/IGNITE-3592>
>>> > >>> >
>>> > >>> > I developed a solution on BinaryMarshaller-level, but
reviewer
>>> has
>>> > >>> rejected
>>> > >>> > it.
>>> > >>> >
>>> > >>> > Let's continue discussion of task goals and solution design.
>>> > >>> > As I understood that, the main goal of this task is to
store
>>> data in
>>> > >>> > compressed form.
>>> > >>> > This is what I need from Ignite as its user. Compression
provides
>>> > >>> economy
>>> > >>> > on
>>> > >>> > servers.
>>> > >>> > We can store more data on same servers at the cost of
increasing
>>> CPU
>>> > >>> > utilization.
>>> > >>> >
>>> > >>> > I'm researching a possibility of implementation of compression
>>> at the
>>> > >>> > cache-level.
>>> > >>> >
>>> > >>> > Any thoughts?
>>> > >>> >
>>> > >>> > --
>>> > >>> > Best regards,
>>> > >>> > Vyacheslav
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > --
>>> > >>> > View this message in context: http://apache-ignite-
>>> > >>> > developers.2346864.n4.nabble.com/Data-compression-in-
>>> > >>> > Ignite-2-0-tp10099p16317.html
>>> > >>> > Sent from the Apache Ignite Developers mailing list archive
at
>>> > >>> Nabble.com.
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Alexey Kuznetsov
>>> > >>>
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Best Regards, Vyacheslav
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards, Vyacheslav
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Vyacheslav
>>> >
>>>
>>
>>
>>
>> --
>> Best Regards, Vyacheslav
>>
>
>
>
> --
> Best Regards, Vyacheslav
>



-- 
Best Regards, Vyacheslav

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message