ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Transport compression (not store compression)
Date Thu, 23 Nov 2017 15:31:26 GMT
Nikita,

Sounds like a good plan. Please share the design details prior getting down to the implementation.

—
Denis

> On Nov 23, 2017, at 4:38 AM, Nikita Amelchev <nsamelchev@gmail.com> wrote:
> 
> Hi Igniters!
> 
> I’m working on the similar feature for my own project.
> I would like to suggest use in-line compression and write encoded bytes in
> network channel by bytes array buffer. It allows us avoiding expensive
> memory allocation.
> The described design may be implemented in TcpCommunicationSpi level. We
> can introduce pluggable compressor on TCP level where we will be able to
> describe our compression strategy, for example, exclude some small messages
> and many other.
> 
> If the community doesn't mind I will file the ticket and will start
> implementing it.
> Any thoughts?
> 
> 2017-11-23 12:06 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
> 
>> Denis,
>> 
>> Regarding zipped marshaller - this would be inefficient, because
>> compression rate will be lower.
>> 
>> On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda <dmagda@apache.org> wrote:
>> 
>>> Nikita,
>>> 
>>> Your solution sounds reasonable from the first glance. However, the
>>> communication layer processes a dozen of small system messages that
>> should
>>> be excluded from the compression. Guess, that we will spend more time on
>>> compressing/decompressing them thus diminishing the positive effect of
>> the
>>> compression.
>>> 
>>> Alexey K., Vladimir O.,
>>> 
>>> What if we create Zip version of the binary marshaller the same way we
>>> implemented GridClientZipOptimizedMarshaller?
>>> 
>>> —
>>> Denis
>>> 
>>>> On Nov 22, 2017, at 5:36 AM, Alexey Kuznetsov <akuznetsov@apache.org>
>>> wrote:
>>>> 
>>>> I think it is very useful feature.
>>>> I also have experience when server nodes connected via fast network.
>>>> But client nodes via very slow network.
>>>> 
>>>> I implemeted GridClientZipOptimizedMarshaller and that solved my
>> issue.
>>>> But this marshaller works only with old
>>>> and org.apache.ignite.internal.client.GridClient and has a lot of
>>>> limitations.
>>>> But compression was about 6-20x times.
>>>> 
>>>> We need a solution for Ignite 2.x and client nodes.
>>>> 
>>>> 
>>>> On Wed, Nov 22, 2017 at 7:48 PM, Nikita Amelchev <nsamelchev@gmail.com
>>> 
>>>> wrote:
>>>> 
>>>>> Hello, Igniters!
>>>>> 
>>>>> I think it is a useful feature. I suggest to implement it to
>>> communication
>>>>> SPI like SSL encryption implemented. I have experience with this
>> feature
>>>>> and I can try to develop it.
>>>>> 
>>>>> 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin <
>> kukushkinalexey@gmail.com
>>>> :
>>>>> 
>>>>>> Forwarding to DEV list: Ignite developers, could you please share
>> your
>>>>>> thoughts on how hard it is to extend Ignite to compress data on the
>>>>>> network.
>>>>>> 
>>>>>> On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) <
>>>>>> gordon.reid@ninemilefinancial.com> wrote:
>>>>>> 
>>>>>>> Hi Igniters,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I see there is a lot of discussion in certain threads about
>>>>> compression.
>>>>>>> This seems to have diverged into conversations about object versus
>>>>> field
>>>>>>> compression, and even throwing encryption into the mix. For my
use
>>>>> case,
>>>>>> I
>>>>>>> am not interested in compressing the cache stored in memory,
I have
>>>>>> plenty
>>>>>>> of memory for my application. What I don’t have is a good network.
I
>>>>>> have a
>>>>>>> high latency, low bandwidth network between my C# ignite client
and
>> my
>>>>>> Java
>>>>>>> ignite server. I only want to compress data when it is sent over
the
>>>>>>> network to remote nodes. It should be stored in the local memory
>>>>>>> uncompressed. How can we achive this? Can the TcpCommunicationSpi
>>>>> support
>>>>>>> compression?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Gordon.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> This email and any attachments are proprietary & confidential
and
>> are
>>>>>>> intended solely for the use of the individuals to whom it is
>>> addressed.
>>>>>> Any
>>>>>>> views or opinions expressed are solely for those of the author
and
>> do
>>>>> not
>>>>>>> necessarily reflect those of Nine Mile Financial Pty. Limited.
If
>> you
>>>>>> have
>>>>>>> received this email in error, please let us know immediately
by
>> reply
>>>>>> email
>>>>>>> and delete from your system. Nine Mile Financial Pty. Limited.
ABN:
>>> 346
>>>>>>> 1349 0252
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Best regards,
>>>>>> Alexey
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Best wishes,
>>>>> Amelchev Nikita
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alexey Kuznetsov
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Best wishes,
> Amelchev Nikita


Mime
View raw message