ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Data compression in Ignite 2.0
Date Tue, 06 Jun 2017 07:59:02 GMT
Vladimir, I am not sure how to interpret the graphs? What are we looking at?

On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <daradurvs@gmail.com>
wrote:

> Hi, Igniters.
>
> I've prepared some benchmarking. Results [1].
>
> And I've prepared the evaluation in the form of diagrams [2].
>
> I hope that helps to interest the community and accelerates a reaction to
> this improvment :)
>
> [1]
> https://github.com/daradurvs/ignite-compression/tree/
> master/src/main/resources/result
> [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view
>
>
>
> 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>
> > Guys, any thoughts?
> >
> > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
> >
> >> Hi guys,
> >>
> >> I've prepared the PR to show my idea.
> >> https://github.com/apache/ignite/pull/1951/files
> >>
> >> About querying - I've just copied existing tests and have annotated the
> >> testing data.
> >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> >> f4058141d059bb577e75244764
> >>
> >> It means fields which will be marked by @BinaryCompression will be
> >> compressed at marshalling via BinaryMarshaller.
> >>
> >> This solution has no effect on existing data or project architecture.
> >>
> >> I'll be glad to see your thougths.
> >>
> >>
> >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
> >>
> >>> Dmitriy,
> >>>
> >>> I have ready prototype. I want to show it.
> >>> It is always easier to discuss on example.
> >>>
> >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
> >>>
> >>>> Vyacheslav,
> >>>>
> >>>> I think it is a bit premature to provide a PR without getting a
> >>>> community
> >>>> consensus on the dev list. Please allow some time for the community
to
> >>>> respond.
> >>>>
> >>>> D.
> >>>>
> >>>> On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
> >>>> daradurvs@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > I created the ticket: https://issues.apache.org/jira
> >>>> /browse/IGNITE-5226
> >>>> >
> >>>> > I'll prepare a PR with described solution in couple of days.
> >>>> >
> >>>> > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com
> >:
> >>>> >
> >>>> > > Hi, Igniters!
> >>>> > >
> >>>> > > Apache 2.0 is released.
> >>>> > >
> >>>> > > Let's continue the discussion about a compression design.
> >>>> > >
> >>>> > > At the moment, I found only one solution which is compatible
with
> >>>> > querying
> >>>> > > and indexing, this is per-objects-field compression.
> >>>> > > Per-fields compression means that metadata (a header) of an
object
> >>>> won't
> >>>> > > be compressed, only serialized values of an object fields
(in
> bytes
> >>>> array
> >>>> > > form) will be compressed.
> >>>> > >
> >>>> > > This solution have some contentious issues:
> >>>> > > - small values, like primitives and short arrays - there isn't
> >>>> sense to
> >>>> > > compress them;
> >>>> > > - there is no possible to use compression with java-predefined
> >>>> types;
> >>>> > >
> >>>> > > We can provide an annotation, @IgniteCompression - for example,
> >>>> which can
> >>>> > > be used by users for marking fields to compress.
> >>>> > >
> >>>> > > Any thoughts?
> >>>> > >
> >>>> > > Maybe someone already have ready design?
> >>>> > >
> >>>> > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <
> daradurvs@gmail.com
> >>>> >:
> >>>> > >
> >>>> > >> Alexey,
> >>>> > >>
> >>>> > >> Yes, I've read it.
> >>>> > >>
> >>>> > >> Ok, let's discuss about public API design.
> >>>> > >>
> >>>> > >> I think we need to add some a configure entity to
> >>>> CacheConfiguration,
> >>>> > >> which will contain the Compressor interface implementation
and
> some
> >>>> > usefull
> >>>> > >> parameters.
> >>>> > >> Or maybe to provide a BinaryMarshaller decorator, which
will be
> >>>> compress
> >>>> > >> data after marshalling.
> >>>> > >>
> >>>> > >>
> >>>> > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov <
> akuznetsov@apache.org
> >>>> >:
> >>>> > >>
> >>>> > >>> Vyacheslav,
> >>>> > >>>
> >>>> > >>> Did you read initial discussion [1] about compression?
> >>>> > >>> As far as I remember we agreed to add only some "top-level"
API
> in
> >>>> > order
> >>>> > >>> to
> >>>> > >>> provide a way for
> >>>> > >>> Ignite users to inject some sort of custom compression.
> >>>> > >>>
> >>>> > >>>
> >>>> > >>> [1]
> >>>> > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c
> >>>> > >>> ompression-in-Ignite-2-0-td10099.html
> >>>> > >>>
> >>>> > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs <daradurvs@gmail.com
> >
> >>>> > wrote:
> >>>> > >>>
> >>>> > >>> > Hi Igniters!
> >>>> > >>> >
> >>>> > >>> > I am interested in this task.
> >>>> > >>> > Provide some kind of pluggable compression SPI
support
> >>>> > >>> > <https://issues.apache.org/jira/browse/IGNITE-3592>
> >>>> > >>> >
> >>>> > >>> > I developed a solution on BinaryMarshaller-level,
but reviewer
> >>>> has
> >>>> > >>> rejected
> >>>> > >>> > it.
> >>>> > >>> >
> >>>> > >>> > Let's continue discussion of task goals and solution
design.
> >>>> > >>> > As I understood that, the main goal of this task
is to store
> >>>> data in
> >>>> > >>> > compressed form.
> >>>> > >>> > This is what I need from Ignite as its user.
Compression
> >>>> provides
> >>>> > >>> economy
> >>>> > >>> > on
> >>>> > >>> > servers.
> >>>> > >>> > We can store more data on same servers at the
cost of
> >>>> increasing CPU
> >>>> > >>> > utilization.
> >>>> > >>> >
> >>>> > >>> > I'm researching a possibility of implementation
of compression
> >>>> at the
> >>>> > >>> > cache-level.
> >>>> > >>> >
> >>>> > >>> > Any thoughts?
> >>>> > >>> >
> >>>> > >>> > --
> >>>> > >>> > Best regards,
> >>>> > >>> > Vyacheslav
> >>>> > >>> >
> >>>> > >>> >
> >>>> > >>> >
> >>>> > >>> >
> >>>> > >>> > --
> >>>> > >>> > View this message in context: http://apache-ignite-
> >>>> > >>> > developers.2346864.n4.nabble.com/Data-compression-in-
> >>>> > >>> > Ignite-2-0-tp10099p16317.html
> >>>> > >>> > Sent from the Apache Ignite Developers mailing
list archive at
> >>>> > >>> Nabble.com.
> >>>> > >>> >
> >>>> > >>>
> >>>> > >>>
> >>>> > >>>
> >>>> > >>> --
> >>>> > >>> Alexey Kuznetsov
> >>>> > >>>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >> --
> >>>> > >> Best Regards, Vyacheslav
> >>>> > >>
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > > Best Regards, Vyacheslav
> >>>> > >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Best Regards, Vyacheslav
> >>>> >
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Vyacheslav
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav
> >>
> >
> >
> >
> > --
> > Best Regards, Vyacheslav
> >
>
>
>
> --
> Best Regards, Vyacheslav
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message