Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
From: Andrey Kornev <andrewkornev@hotmail.com>
To: "dev@ignite.apache.org" <dev@ignite.apache.org>
Subject: Re: Data compression in Ignite 2.0
Thread-Topic: Data compression in Ignite 2.0
Thread-Index: AQHR5j3IhhyZevfX7kSX0KMTvD2yO6AovO2AgAAFzgCAANp3gIAAAkoAgAABPoCAAD63nIAAyLqAgAEQ40Q=
Date: Wed, 27 Jul 2016 04:53:10 +0000
Message-ID: <DM2PR0701MB1230BB9221156D6F1318CA33CF0F0@DM2PR0701MB1230.namprd07.prod.outlook.com>
References: <CALH+G9qs+jAr3Q+K+40y_A390ixPfsfZ4PdSNaEhRAznhS+C+A@mail.gmail.com>
 <CA+eZwrHRyaLV0--rWZ6uf=ynQPX_KWbSd7=FL8Xaoy4gfa_H_g@mail.gmail.com>
 <CALH+G9roN0b6dWv6fuqE-gaJk6vKtcGWrnqDyNru3jB=AXX=2g@mail.gmail.com>
 <CAMbfdjasRph0wbAe7KBniTe-agWdrG1-YA25XwTAcGmawkCsyA@mail.gmail.com>
 <CALV17QrUX1DRNP8KYH5Hz5C5Sv7QmFifD7qar1jaZLQQhcPA-A@mail.gmail.com>
 <CA+0=VoUG2-=L2CxnCykzJnM7OY8pYhw5jMietVNdE7+DaPuRCg@mail.gmail.com>
 <DM2PR0701MB1230B43298AEC92D7C1E051BCF0E0@DM2PR0701MB1230.namprd07.prod.outlook.com>,<CALH+G9pg5t+Z5NNpaSTpiBx927PZzRjgey=+G4+KxMiFmHBcSA@mail.gmail.com>
In-Reply-To: <CALH+G9pg5t+Z5NNpaSTpiBx927PZzRjgey=+G4+KxMiFmHBcSA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: SoftFail (protection.outlook.com: domain of transitioning
 hotmail.com discourages use of 10.152.74.51 as permitted sender)
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [pnLog1fUFw8d8ur3Gwe9CPQazkM1ALqH]
x-eopattributedmessage: 0
x-forefront-antispam-report: CIP:10.152.74.51;IPV:NLI;CTRY:;EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:CY1NAM02HT256;H:DM2PR0701MB1230.namprd07.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-microsoft-exchange-diagnostics: 1;CY1NAM02HT256;6:BR6uuYbiRfO5/tcQCNT7V4SCb0hTN3ADkEkYyBz1UsQPM6SQkq+JnRcynTEJqA9Zv9cIfxPxKLW3XEn/F42fZO4FDWEJce9IBP2K/+zHps/LPPSZ+q9P0UwZPyxDzlBjSFTfGqc1fOIP30ib05IEC14hPHlrKXxem4Qkjylz3W56rYKi98832Pk/UpzTbOXH2xRZnWkyaLBikEKpj+6jVRgfjHMMjz5i5GtRionQ8wGPIm6BsOpjh9o57z2f2ZJVLgVfXqGnxGf7xKli4s1T2yKbJ2wu8Rg/xey7jqNfJ/YQTf8GG+iEArENXu3l/oGo;5:xKUaQAdrrG9FDGaT/w9B3xnOcD0/tEkziFCTQ2/tVIk8asXe+Ip/rNF/DIlawyHpJAOMpYLBDTJOS6TGKAaoiY2ylIi5+NVST1hk74AaOFx0TZfDq2lAeCM1pD0Nv4kMohddSe/ZRFxnl4zoDmgBvA==;24:fvcQAYM5ON4b7JOsZiEJWDMQJIoynkoMEYGd4Ae3E4lTieOYWgaG2iJfNXbqkvVVT79lGiSG3reV/62Db/kGWMZcDGCjE4l0kBifRSDxUyE=;7:EIwyCX7DWPyMDqGWU2rhEUU6xqH6if9f9+exczO8oMGhIryDvGq96GtjipkkzE9ruH5y/YlG6OfNq6zA2XX+c9c9FItrcT27qt/9FZn+ErGWkKFnye7V+r0q962sdjHc+Tqn/U5CjwoyYDG8Ksr6ABDubqv/vIWUOo7t6H0F2WKp/WO2JrZfgf0pIPXPJyrBoY+u2zl7HGVvpQYn9FIlXeakk/FAjiUIKGFqFTvbdqTgpOehfImtPH7uBx2FW+sr
x-ms-office365-filtering-correlation-id: 78b3eebc-88c3-4d06-4679-08d3b5d9e8a2
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(1601124038)(1601125047);SRVR:CY1NAM02HT256;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:CY1NAM02HT256;BCL:0;PCL:0;RULEID:;SRVR:CY1NAM02HT256;
x-forefront-prvs: 0016DEFF96
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_DM2PR0701MB1230BB9221156D6F1318CA33CF0F0DM2PR0701MB1230_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jul 2016 04:53:10.1058
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1NAM02HT256
X-OriginalArrivalTime: 27 Jul 2016 04:53:13.0139 (UTC) FILETIME=[C7C03830:01D1E7C2]
archived-at: Wed, 27 Jul 2016 04:53:26 -0000

--_000_DM2PR0701MB1230BB9221156D6F1318CA33CF0F0DM2PR0701MB1230_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dictionary compression requires some knowledge about data being compressed.=
 For example, for numeric types a range of values must be known so that the=
 dictionary can be generated. For strings, the number of unique values of t=
he column is the key piece of input into the dictionary generation.
SAP HANA is a column-based database system: it stores the fields of the dat=
a tuple individually using the best compression for the given data type and=
 the particular set of values. HANA has been specifically built as a genera=
l purpose database, rather than as an afterthought layer on top of an alrea=
dy existing distributed cache.
On the other hand, Ignite is a distributed cache implementation (a pretty g=
ood one!) that in general requires no schema and stores its data in the row=
-based fashion. Its current design doesn't land itself readily to the kind =
of optimizations HANA provides out of the box.
For the curios types among us, the implementation details of HANA are well =
documented in "In-memory Data Management", by Hasso Plattner & Alexander Ze=
ier.
Cheers
Andrey
_____________________________
From: Alexey Kuznetsov <akuznetsov@gridgain.com<mailto:akuznetsov@gridgain.=
com>>
Sent: Tuesday, July 26, 2016 5:36 AM
Subject: Re: Data compression in Ignite 2.0
To: <dev@ignite.apache.org<mailto:dev@ignite.apache.org>>


Sergey Kozlov wrote:
>> For approach 1: Put a large object into a partition cache will
force to update
the dictionary placed on replication cache. It may be time-expense
operation.
The dictionary will be built only once. And we could control what should be
put into dictionary, for example, we could check min and max size and
decide - put value to dictionary or not.

>> Approach 2-3 are make sense for rare cases as Sergi commented.
But it is better at least have a possibility to plug user code for
compression than not to have it at all.

>> Also I see a danger of OOM if we've got high compression level and try
to restore original value in memory.
We could easily get OOM with many other operations right now without
compression, I think it is not an issue, we could add a NOTE to
documentation about such possibility.

Andrey Kornev wrote:
>> ... in general I think compression is a great data. The cleanest way to
achieve that would be to just make it possible to chain the marshallers...
I think it is also good idea. And looks like it could be used for
compression with some sort of ZIP algorithm, but how to deal with
compression by dictionary substitution?
We need to build dictionary first. Any ideas?

Nikita Ivanov wrote:
>> SAP Hana does the compression by 1) compressing SQL parameters before
execution...
Looks interesting, but my initial point was about compression of cache
data, not SQL queries.
My idea was to make compression transparent for SQL engine when it will
lookup for data.

But idea of compressing SQL queries result looks very interesting, because
it is known fact, that SQL engine could consume quite a lot of heap for
storing result sets.
I think this should be discussed in separate thread.

Just for you information, in first message I mentioned that DB2 has
compression by dictionary and according to them it is possible to
compress usual data to 50-80%.
I have some experience with DB2 and can confirm this.

--
Alexey Kuznetsov


--_000_DM2PR0701MB1230BB9221156D6F1318CA33CF0F0DM2PR0701MB1230_--