Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 25 Jan 2012 17:34:41 +0000 (UTC)
From: "Mikhail Bautin (Commented) (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <51191861.77031.1327512881381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <657671452.46177.1313609127186.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues
  (aka delta encoding / prefix compression)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-4218?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1319=
3148#comment-13193148 ]=20

Mikhail Bautin commented on HBASE-4218:
---------------------------------------

Re-running unit tests that failed on Jenkins:

Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 52, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 181.919 se=
c
Running org.apache.hadoop.hbase.client.TestAdmin
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 195.194 se=
c
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 223.405 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.48 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.561 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.289 sec
Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.362 sec

Results :

Tests run: 122, Failures: 0, Errors: 0, Skipped: 3

               =20
> Data Block Encoding of KeyValues  (aka delta encoding / prefix compressio=
n)
> -------------------------------------------------------------------------=
--
>
>                 Key: HBASE-4218
>                 URL: https://issues.apache.org/jira/browse/HBASE-4218
>             Project: HBase
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.94.0
>            Reporter: Jacek Migdal
>            Assignee: Mikhail Bautin
>              Labels: compression
>             Fix For: 0.94.0
>
>         Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 00=
01-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, D447.=
1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.1=
4.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, D447.1=
9.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, D447.23=
.patch, D447.24.patch, D447.25.patch, D447.3.patch, D447.4.patch, D447.5.pa=
tch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, Data-block-enc=
oding-2011-12-23.patch, Delta-encoding-2012-01-17_11_09_09.patch, Delta-enc=
oding-2012-01-25_00_45_29.patch, Delta-encoding.patch-2011-12-22_11_52_07.p=
atch, Delta-encoding.patch-2012-01-05_15_16_43.patch, Delta-encoding.patch-=
2012-01-05_16_31_44.patch, Delta-encoding.patch-2012-01-05_16_31_44_copy.pa=
tch, Delta-encoding.patch-2012-01-05_18_50_47.patch, Delta-encoding.patch-2=
012-01-07_14_12_48.patch, Delta-encoding.patch-2012-01-13_12_20_07.patch, D=
elta_encoding_with_memstore_TS.patch, open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually ver=
y similar. Because of that, it is possible to design better compression tha=
n general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save m=
emory in cache as well as speeding seeks within HFileBlocks. It should impr=
ove performance a lot, if key lengths are larger than value lengths. For ex=
ample, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length =3D ~ 90 bytes , value length =3D =
8 bytes) shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio t=
han LZO). Moreover, it should allow far more efficient seeking which should=
 improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of th=
e savings are due to prefix compression, int128 encoding, timestamp diffs a=
nd bitfields to avoid duplication. That way, comparisons of compressed data=
 can be much faster than a byte comparator (thanks to prefix compression an=
d bitfields).
> In order to implement it in HBase two important changes in design will be=
 needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seekin=
g and iterating; access to uncompressed buffer in HFileBlock will have bad =
performance
> -extend comparators to support comparison assuming that N first bytes are=
 equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=3DRe+prefix+com=
pression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira