ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vyacheslav Daradur <daradu...@gmail.com>
Subject Re: Data compression in Ignite 2.0
Date Tue, 06 Jun 2017 12:08:50 GMT
I wish to note, that the results of benchmarking shows metrics from
stress-testing.
I mean in real scenarios, for example business operations, which take
milliseconds or seсonds, increase in time of put-get-operation will be
insignificant.

2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:

> Conclusion:
> Provided solution allows reduce size of an object in IgniteCache at the
> cost of throughput reduction (small - in some cases), it depends on part of
> object which will be compressed and compression algorithm.
> I mean, we can make more effective use of memory, and in some cases it can
> reduce loading of the interconnect. (replication, rebalancing)
>
> Especially, it will be particularly useful for object's fields which are
> large text (>~ 250 bytes) and can be effectively compressed.
>
> 2017-06-06 12:00 GMT+03:00 Антон Чураев <churaev.an@gmail.com>:
>
>> Vyacheslav, thank you! But could you please provide a conclusions or
>> proposals based on this benchmarks?
>>
>> 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>>
>> > Dmitry,
>> >
>> > Excel-pages:
>> >
>> > 1). "Compression ratio (2)" - shows object size, with compression and
>> > without compression. (Conditions: literal text)
>> > 1st graph shows compression ratios of using different compression
>> algrithms
>> > depending on size of compressed field.
>> > 2nd graph shows evaluation of size of objects depending on sizes and
>> > compression algorithms.
>> >
>> > 2). "Compression ratio (1)" - shows object size, with compression and
>> > without compression. (Conditions:  badly compressed character sequence)
>> > 1st graph shows compression ratios of using different compression
>> > algrithms depending on size of compressed field.
>> > 2nd graph shows evaluation of size of objects depending on sizes and
>> > compression algorithms.
>> >
>> > 3) 'put-avg" - shows average time of the "put" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 4) 'put-thrpt" - shows throughput of the "put" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 5) 'get-avg" - shows average time of the "get" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 6) 'get-thrpt" - shows throughput of the "get" operation depending on
>> size
>> > and compression algorithms.
>> >
>> >
>> >
>> >
>> > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
>> >
>> > > Vladimir, I am not sure how to interpret the graphs? What are we
>> looking
>> > > at?
>> > >
>> > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <
>> daradurvs@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Hi, Igniters.
>> > > >
>> > > > I've prepared some benchmarking. Results [1].
>> > > >
>> > > > And I've prepared the evaluation in the form of diagrams [2].
>> > > >
>> > > > I hope that helps to interest the community and accelerates a
>> reaction
>> > to
>> > > > this improvment :)
>> > > >
>> > > > [1]
>> > > > https://github.com/daradurvs/ignite-compression/tree/
>> > > > master/src/main/resources/result
>> > > > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk
>> /view
>> > > >
>> > > >
>> > > >
>> > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur <daradurvs@gmail.com>:
>> > > >
>> > > > > Guys, any thoughts?
>> > > > >
>> > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <
>> daradurvs@gmail.com>:
>> > > > >
>> > > > >> Hi guys,
>> > > > >>
>> > > > >> I've prepared the PR to show my idea.
>> > > > >> https://github.com/apache/ignite/pull/1951/files
>> > > > >>
>> > > > >> About querying - I've just copied existing tests and have
>> annotated
>> > > the
>> > > > >> testing data.
>> > > > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
>> > > > >> f4058141d059bb577e75244764
>> > > > >>
>> > > > >> It means fields which will be marked by @BinaryCompression
will
>> be
>> > > > >> compressed at marshalling via BinaryMarshaller.
>> > > > >>
>> > > > >> This solution has no effect on existing data or project
>> > architecture.
>> > > > >>
>> > > > >> I'll be glad to see your thougths.
>> > > > >>
>> > > > >>
>> > > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <
>> daradurvs@gmail.com
>> > >:
>> > > > >>
>> > > > >>> Dmitriy,
>> > > > >>>
>> > > > >>> I have ready prototype. I want to show it.
>> > > > >>> It is always easier to discuss on example.
>> > > > >>>
>> > > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <
>> > dsetrakyan@apache.org
>> > > >:
>> > > > >>>
>> > > > >>>> Vyacheslav,
>> > > > >>>>
>> > > > >>>> I think it is a bit premature to provide a PR without
getting a
>> > > > >>>> community
>> > > > >>>> consensus on the dev list. Please allow some time
for the
>> > community
>> > > to
>> > > > >>>> respond.
>> > > > >>>>
>> > > > >>>> D.
>> > > > >>>>
>> > > > >>>> On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur
<
>> > > > >>>> daradurvs@gmail.com>
>> > > > >>>> wrote:
>> > > > >>>>
>> > > > >>>> > I created the ticket: https://issues.apache.org/jira
>> > > > >>>> /browse/IGNITE-5226
>> > > > >>>> >
>> > > > >>>> > I'll prepare a PR with described solution in
couple of days.
>> > > > >>>> >
>> > > > >>>> > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur
<
>> > > daradurvs@gmail.com
>> > > > >:
>> > > > >>>> >
>> > > > >>>> > > Hi, Igniters!
>> > > > >>>> > >
>> > > > >>>> > > Apache 2.0 is released.
>> > > > >>>> > >
>> > > > >>>> > > Let's continue the discussion about a compression
design.
>> > > > >>>> > >
>> > > > >>>> > > At the moment, I found only one solution
which is
>> compatible
>> > > with
>> > > > >>>> > querying
>> > > > >>>> > > and indexing, this is per-objects-field
compression.
>> > > > >>>> > > Per-fields compression means that metadata
(a header) of an
>> > > object
>> > > > >>>> won't
>> > > > >>>> > > be compressed, only serialized values of
an object fields
>> (in
>> > > > bytes
>> > > > >>>> array
>> > > > >>>> > > form) will be compressed.
>> > > > >>>> > >
>> > > > >>>> > > This solution have some contentious issues:
>> > > > >>>> > > - small values, like primitives and short
arrays - there
>> isn't
>> > > > >>>> sense to
>> > > > >>>> > > compress them;
>> > > > >>>> > > - there is no possible to use compression
with
>> java-predefined
>> > > > >>>> types;
>> > > > >>>> > >
>> > > > >>>> > > We can provide an annotation, @IgniteCompression
- for
>> > example,
>> > > > >>>> which can
>> > > > >>>> > > be used by users for marking fields to
compress.
>> > > > >>>> > >
>> > > > >>>> > > Any thoughts?
>> > > > >>>> > >
>> > > > >>>> > > Maybe someone already have ready design?
>> > > > >>>> > >
>> > > > >>>> > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur
<
>> > > > daradurvs@gmail.com
>> > > > >>>> >:
>> > > > >>>> > >
>> > > > >>>> > >> Alexey,
>> > > > >>>> > >>
>> > > > >>>> > >> Yes, I've read it.
>> > > > >>>> > >>
>> > > > >>>> > >> Ok, let's discuss about public API
design.
>> > > > >>>> > >>
>> > > > >>>> > >> I think we need to add some a configure
entity to
>> > > > >>>> CacheConfiguration,
>> > > > >>>> > >> which will contain the Compressor interface
implementation
>> > and
>> > > > some
>> > > > >>>> > usefull
>> > > > >>>> > >> parameters.
>> > > > >>>> > >> Or maybe to provide a BinaryMarshaller
decorator, which
>> will
>> > be
>> > > > >>>> compress
>> > > > >>>> > >> data after marshalling.
>> > > > >>>> > >>
>> > > > >>>> > >>
>> > > > >>>> > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov
<
>> > > > akuznetsov@apache.org
>> > > > >>>> >:
>> > > > >>>> > >>
>> > > > >>>> > >>> Vyacheslav,
>> > > > >>>> > >>>
>> > > > >>>> > >>> Did you read initial discussion
[1] about compression?
>> > > > >>>> > >>> As far as I remember we agreed
to add only some
>> "top-level"
>> > > API
>> > > > in
>> > > > >>>> > order
>> > > > >>>> > >>> to
>> > > > >>>> > >>> provide a way for
>> > > > >>>> > >>> Ignite users to inject some sort
of custom compression.
>> > > > >>>> > >>>
>> > > > >>>> > >>>
>> > > > >>>> > >>> [1]
>> > > > >>>> > >>> http://apache-ignite-developers.2346864.n4.nabble.
>> > com/Data-c
>> > > > >>>> > >>> ompression-in-Ignite-2-0-td10099.html
>> > > > >>>> > >>>
>> > > > >>>> > >>> On Mon, Apr 10, 2017 at 2:19 PM,
daradurvs <
>> > > daradurvs@gmail.com
>> > > > >
>> > > > >>>> > wrote:
>> > > > >>>> > >>>
>> > > > >>>> > >>> > Hi Igniters!
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > I am interested in this task.
>> > > > >>>> > >>> > Provide some kind of pluggable
compression SPI support
>> > > > >>>> > >>> > <https://issues.apache.org/jira/browse/IGNITE-3592>
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > I developed a solution on
BinaryMarshaller-level, but
>> > > reviewer
>> > > > >>>> has
>> > > > >>>> > >>> rejected
>> > > > >>>> > >>> > it.
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > Let's continue discussion
of task goals and solution
>> > design.
>> > > > >>>> > >>> > As I understood that, the
main goal of this task is to
>> > store
>> > > > >>>> data in
>> > > > >>>> > >>> > compressed form.
>> > > > >>>> > >>> > This is what I need from Ignite
as its user.
>> Compression
>> > > > >>>> provides
>> > > > >>>> > >>> economy
>> > > > >>>> > >>> > on
>> > > > >>>> > >>> > servers.
>> > > > >>>> > >>> > We can store more data on
same servers at the cost of
>> > > > >>>> increasing CPU
>> > > > >>>> > >>> > utilization.
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > I'm researching a possibility
of implementation of
>> > > compression
>> > > > >>>> at the
>> > > > >>>> > >>> > cache-level.
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > Any thoughts?
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > --
>> > > > >>>> > >>> > Best regards,
>> > > > >>>> > >>> > Vyacheslav
>> > > > >>>> > >>> >
>> > > > >>>> > >>> >
>> > > > >>>> > >>> >
>> > > > >>>> > >>> >
>> > > > >>>> > >>> > --
>> > > > >>>> > >>> > View this message in context:
http://apache-ignite-
>> > > > >>>> > >>> > developers.2346864.n4.nabble.com/Data-compression-in-
>> > > > >>>> > >>> > Ignite-2-0-tp10099p16317.html
>> > > > >>>> > >>> > Sent from the Apache Ignite
Developers mailing list
>> > archive
>> > > at
>> > > > >>>> > >>> Nabble.com.
>> > > > >>>> > >>> >
>> > > > >>>> > >>>
>> > > > >>>> > >>>
>> > > > >>>> > >>>
>> > > > >>>> > >>> --
>> > > > >>>> > >>> Alexey Kuznetsov
>> > > > >>>> > >>>
>> > > > >>>> > >>
>> > > > >>>> > >>
>> > > > >>>> > >>
>> > > > >>>> > >> --
>> > > > >>>> > >> Best Regards, Vyacheslav
>> > > > >>>> > >>
>> > > > >>>> > >
>> > > > >>>> > >
>> > > > >>>> > >
>> > > > >>>> > > --
>> > > > >>>> > > Best Regards, Vyacheslav
>> > > > >>>> > >
>> > > > >>>> >
>> > > > >>>> >
>> > > > >>>> >
>> > > > >>>> > --
>> > > > >>>> > Best Regards, Vyacheslav
>> > > > >>>> >
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards, Vyacheslav
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Best Regards, Vyacheslav
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Best Regards, Vyacheslav
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best Regards, Vyacheslav
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav
>> >
>>
>>
>>
>> --
>>
>> Best Regards, Anton Churaev
>>
>
>
>
> --
> Best Regards, Vyacheslav
>



-- 
Best Regards, Vyacheslav

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message