Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
From: Denis Magda <dmagda@apache.org>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: Transport compression (not store compression)
Date: Thu, 23 Nov 2017 07:31:26 -0800
References: <ME1PR01MB1585628B38EDB7DDFD220459F6200@ME1PR01MB1585.ausprd01.prod.outlook.com>
 <CA+vFNSoRYu9W8eQ3M9SUzTcRfEB2Cpt8mBTCAGYvvYywi_CQKw@mail.gmail.com>
 <CAFtZiX3Zhxavz2nXg3E5J5a7hYrnGNHWjW1G7Tb2Z+Wq8Dq8LA@mail.gmail.com>
 <CALH+G9rTxE2GWP71UGJpCh6NM2a1wtkgzRDCcgF2=3OxCuNfig@mail.gmail.com>
 <E702C293-8B10-4066-9571-041DC6E789F1@apache.org>
 <CAGt6o8nYExOovvFqUqSnv1EEgn=9rQGUofMkx3V7hAoU8CjdbA@mail.gmail.com>
 <CAFtZiX14FD48tjUiYPna7K6gvYsjG5t8O0ObEE-hvpR-zzM2pQ@mail.gmail.com>
To: dev@ignite.apache.org
In-Reply-To: <CAFtZiX14FD48tjUiYPna7K6gvYsjG5t8O0ObEE-hvpR-zzM2pQ@mail.gmail.com>
Message-Id: <637C9F25-9FF2-4EA5-8D2F-C6272F95BEBF@apache.org>
archived-at: Thu, 23 Nov 2017 15:31:30 -0000

Nikita,

Sounds like a good plan. Please share the design details prior getting =
down to the implementation.

=E2=80=94
Denis

> On Nov 23, 2017, at 4:38 AM, Nikita Amelchev <nsamelchev@gmail.com> =
wrote:
>=20
> Hi Igniters!
>=20
> I=E2=80=99m working on the similar feature for my own project.
> I would like to suggest use in-line compression and write encoded =
bytes in
> network channel by bytes array buffer. It allows us avoiding expensive
> memory allocation.
> The described design may be implemented in TcpCommunicationSpi level. =
We
> can introduce pluggable compressor on TCP level where we will be able =
to
> describe our compression strategy, for example, exclude some small =
messages
> and many other.
>=20
> If the community doesn't mind I will file the ticket and will start
> implementing it.
> Any thoughts?
>=20
> 2017-11-23 12:06 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
>=20
>> Denis,
>>=20
>> Regarding zipped marshaller - this would be inefficient, because
>> compression rate will be lower.
>>=20
>> On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda <dmagda@apache.org> =
wrote:
>>=20
>>> Nikita,
>>>=20
>>> Your solution sounds reasonable from the first glance. However, the
>>> communication layer processes a dozen of small system messages that
>> should
>>> be excluded from the compression. Guess, that we will spend more =
time on
>>> compressing/decompressing them thus diminishing the positive effect =
of
>> the
>>> compression.
>>>=20
>>> Alexey K., Vladimir O.,
>>>=20
>>> What if we create Zip version of the binary marshaller the same way =
we
>>> implemented GridClientZipOptimizedMarshaller?
>>>=20
>>> =E2=80=94
>>> Denis
>>>=20
>>>> On Nov 22, 2017, at 5:36 AM, Alexey Kuznetsov =
<akuznetsov@apache.org>
>>> wrote:
>>>>=20
>>>> I think it is very useful feature.
>>>> I also have experience when server nodes connected via fast =
network.
>>>> But client nodes via very slow network.
>>>>=20
>>>> I implemeted GridClientZipOptimizedMarshaller and that solved my
>> issue.
>>>> But this marshaller works only with old
>>>> and org.apache.ignite.internal.client.GridClient and has a lot of
>>>> limitations.
>>>> But compression was about 6-20x times.
>>>>=20
>>>> We need a solution for Ignite 2.x and client nodes.
>>>>=20
>>>>=20
>>>> On Wed, Nov 22, 2017 at 7:48 PM, Nikita Amelchev =
<nsamelchev@gmail.com
>>>=20
>>>> wrote:
>>>>=20
>>>>> Hello, Igniters!
>>>>>=20
>>>>> I think it is a useful feature. I suggest to implement it to
>>> communication
>>>>> SPI like SSL encryption implemented. I have experience with this
>> feature
>>>>> and I can try to develop it.
>>>>>=20
>>>>> 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin <
>> kukushkinalexey@gmail.com
>>>> :
>>>>>=20
>>>>>> Forwarding to DEV list: Ignite developers, could you please share
>> your
>>>>>> thoughts on how hard it is to extend Ignite to compress data on =
the
>>>>>> network.
>>>>>>=20
>>>>>> On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) <
>>>>>> gordon.reid@ninemilefinancial.com> wrote:
>>>>>>=20
>>>>>>> Hi Igniters,
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> I see there is a lot of discussion in certain threads about
>>>>> compression.
>>>>>>> This seems to have diverged into conversations about object =
versus
>>>>> field
>>>>>>> compression, and even throwing encryption into the mix. For my =
use
>>>>> case,
>>>>>> I
>>>>>>> am not interested in compressing the cache stored in memory, I =
have
>>>>>> plenty
>>>>>>> of memory for my application. What I don=E2=80=99t have is a =
good network. I
>>>>>> have a
>>>>>>> high latency, low bandwidth network between my C# ignite client =
and
>> my
>>>>>> Java
>>>>>>> ignite server. I only want to compress data when it is sent over =
the
>>>>>>> network to remote nodes. It should be stored in the local memory
>>>>>>> uncompressed. How can we achive this? Can the =
TcpCommunicationSpi
>>>>> support
>>>>>>> compression?
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> Thanks,
>>>>>>>=20
>>>>>>> Gordon.
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> This email and any attachments are proprietary & confidential =
and
>> are
>>>>>>> intended solely for the use of the individuals to whom it is
>>> addressed.
>>>>>> Any
>>>>>>> views or opinions expressed are solely for those of the author =
and
>> do
>>>>> not
>>>>>>> necessarily reflect those of Nine Mile Financial Pty. Limited. =
If
>> you
>>>>>> have
>>>>>>> received this email in error, please let us know immediately by
>> reply
>>>>>> email
>>>>>>> and delete from your system. Nine Mile Financial Pty. Limited. =
ABN:
>>> 346
>>>>>>> 1349 0252
>>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> --
>>>>>> Best regards,
>>>>>> Alexey
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> --
>>>>> Best wishes,
>>>>> Amelchev Nikita
>>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> --
>>>> Alexey Kuznetsov
>>>=20
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> Best wishes,
> Amelchev Nikita