ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikita Amelchev <nsamelc...@gmail.com>
Subject Re: Transport compression (not store compression)
Date Tue, 28 Nov 2017 07:39:00 GMT
Hi,
I've filed a ticket [1]. I'll try to share design details in a couple of
days.

1. https://issues.apache.org/jira/browse/IGNITE-7024

2017-11-23 18:31 GMT+03:00 Denis Magda <dmagda@apache.org>:

> Nikita,
>
> Sounds like a good plan. Please share the design details prior getting
> down to the implementation.
>
> —
> Denis
>
> > On Nov 23, 2017, at 4:38 AM, Nikita Amelchev <nsamelchev@gmail.com>
> wrote:
> >
> > Hi Igniters!
> >
> > I’m working on the similar feature for my own project.
> > I would like to suggest use in-line compression and write encoded bytes
> in
> > network channel by bytes array buffer. It allows us avoiding expensive
> > memory allocation.
> > The described design may be implemented in TcpCommunicationSpi level. We
> > can introduce pluggable compressor on TCP level where we will be able to
> > describe our compression strategy, for example, exclude some small
> messages
> > and many other.
> >
> > If the community doesn't mind I will file the ticket and will start
> > implementing it.
> > Any thoughts?
> >
> > 2017-11-23 12:06 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
> >
> >> Denis,
> >>
> >> Regarding zipped marshaller - this would be inefficient, because
> >> compression rate will be lower.
> >>
> >> On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda <dmagda@apache.org> wrote:
> >>
> >>> Nikita,
> >>>
> >>> Your solution sounds reasonable from the first glance. However, the
> >>> communication layer processes a dozen of small system messages that
> >> should
> >>> be excluded from the compression. Guess, that we will spend more time
> on
> >>> compressing/decompressing them thus diminishing the positive effect of
> >> the
> >>> compression.
> >>>
> >>> Alexey K., Vladimir O.,
> >>>
> >>> What if we create Zip version of the binary marshaller the same way we
> >>> implemented GridClientZipOptimizedMarshaller?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On Nov 22, 2017, at 5:36 AM, Alexey Kuznetsov <akuznetsov@apache.org>
> >>> wrote:
> >>>>
> >>>> I think it is very useful feature.
> >>>> I also have experience when server nodes connected via fast network.
> >>>> But client nodes via very slow network.
> >>>>
> >>>> I implemeted GridClientZipOptimizedMarshaller and that solved my
> >> issue.
> >>>> But this marshaller works only with old
> >>>> and org.apache.ignite.internal.client.GridClient and has a lot of
> >>>> limitations.
> >>>> But compression was about 6-20x times.
> >>>>
> >>>> We need a solution for Ignite 2.x and client nodes.
> >>>>
> >>>>
> >>>> On Wed, Nov 22, 2017 at 7:48 PM, Nikita Amelchev <
> nsamelchev@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Hello, Igniters!
> >>>>>
> >>>>> I think it is a useful feature. I suggest to implement it to
> >>> communication
> >>>>> SPI like SSL encryption implemented. I have experience with this
> >> feature
> >>>>> and I can try to develop it.
> >>>>>
> >>>>> 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin <
> >> kukushkinalexey@gmail.com
> >>>> :
> >>>>>
> >>>>>> Forwarding to DEV list: Ignite developers, could you please
share
> >> your
> >>>>>> thoughts on how hard it is to extend Ignite to compress data
on the
> >>>>>> network.
> >>>>>>
> >>>>>> On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) <
> >>>>>> gordon.reid@ninemilefinancial.com> wrote:
> >>>>>>
> >>>>>>> Hi Igniters,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I see there is a lot of discussion in certain threads about
> >>>>> compression.
> >>>>>>> This seems to have diverged into conversations about object
versus
> >>>>> field
> >>>>>>> compression, and even throwing encryption into the mix.
For my use
> >>>>> case,
> >>>>>> I
> >>>>>>> am not interested in compressing the cache stored in memory,
I have
> >>>>>> plenty
> >>>>>>> of memory for my application. What I don’t have is a good
network.
> I
> >>>>>> have a
> >>>>>>> high latency, low bandwidth network between my C# ignite
client and
> >> my
> >>>>>> Java
> >>>>>>> ignite server. I only want to compress data when it is sent
over
> the
> >>>>>>> network to remote nodes. It should be stored in the local
memory
> >>>>>>> uncompressed. How can we achive this? Can the TcpCommunicationSpi
> >>>>> support
> >>>>>>> compression?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Gordon.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This email and any attachments are proprietary & confidential
and
> >> are
> >>>>>>> intended solely for the use of the individuals to whom it
is
> >>> addressed.
> >>>>>> Any
> >>>>>>> views or opinions expressed are solely for those of the
author and
> >> do
> >>>>> not
> >>>>>>> necessarily reflect those of Nine Mile Financial Pty. Limited.
If
> >> you
> >>>>>> have
> >>>>>>> received this email in error, please let us know immediately
by
> >> reply
> >>>>>> email
> >>>>>>> and delete from your system. Nine Mile Financial Pty. Limited.
ABN:
> >>> 346
> >>>>>>> 1349 0252
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>> Alexey
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Best wishes,
> >>>>> Amelchev Nikita
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alexey Kuznetsov
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
>
>


-- 
Best wishes,
Amelchev Nikita

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message