Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 25 Feb 2012 06:09:48 +0000 (UTC)
From: "Hudson (Commented) (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <320310337.19065.1330150188064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <657671452.46177.1313609127186.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues
  (aka delta encoding / prefix compression)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-4218?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1321=
6331#comment-13216331 ]=20

Hudson commented on HBASE-4218:
-------------------------------

Integrated in HBase-TRUNK #2669 (See [https://builds.apache.org/job/HBase-T=
RUNK/2669/])
    [jira] [HBASE-5470] Make DataBlockEncodingTool work correctly with no n=
ative
compression codecs loaded

Summary:
DataBlockEncodingTool was fixed as part of porting data block encoding
(HBASE-4218) to 89-fb
(https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1245291,
https://reviews.facebook.net/D1659). The bug being fixed here appeared when
using GZ as baseline compression codec but not loading native Hadoop librar=
ies,
in which case the compressor instance would be null.

Test Plan:
Run DataBlockEncoding tool with GZ (no native codecs) and LZO (with native
codecs) as baseline (Hadoop-level) compression codecs

Reviewers: JIRA, Kannan, mcorgan, lhofhansl, todd, stack, tedyu

Reviewed By: tedyu

Differential Revision: https://reviews.facebook.net/D1917 (Revision 1293057=
)

     Result =3D SUCCESS
mbautin :=20
Files :=20
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlock=
EncodingTool.java

               =20
> Data Block Encoding of KeyValues  (aka delta encoding / prefix compressio=
n)
> -------------------------------------------------------------------------=
--
>
>                 Key: HBASE-4218
>                 URL: https://issues.apache.org/jira/browse/HBASE-4218
>             Project: HBase
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.94.0
>            Reporter: Jacek Migdal
>            Assignee: Mikhail Bautin
>              Labels: compression
>             Fix For: 0.94.0
>
>         Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 00=
01-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, D1659=
.1.patch, D1659.2.patch, D1659.3.patch, D447.1.patch, D447.10.patch, D447.1=
1.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, D447.1=
6.patch, D447.17.patch, D447.18.patch, D447.19.patch, D447.2.patch, D447.20=
.patch, D447.21.patch, D447.22.patch, D447.23.patch, D447.24.patch, D447.25=
.patch, D447.26.patch, D447.3.patch, D447.4.patch, D447.5.patch, D447.6.pat=
ch, D447.7.patch, D447.8.patch, D447.9.patch, Data-block-encoding-2011-12-2=
3.patch, Delta-encoding-2012-01-17_11_09_09.patch, Delta-encoding-2012-01-2=
5_00_45_29.patch, Delta-encoding-2012-01-25_16_32_14.patch, Delta-encoding.=
patch-2011-12-22_11_52_07.patch, Delta-encoding.patch-2012-01-05_15_16_43.p=
atch, Delta-encoding.patch-2012-01-05_16_31_44.patch, Delta-encoding.patch-=
2012-01-05_16_31_44_copy.patch, Delta-encoding.patch-2012-01-05_18_50_47.pa=
tch, Delta-encoding.patch-2012-01-07_14_12_48.patch, Delta-encoding.patch-2=
012-01-13_12_20_07.patch, Delta_encoding_with_memstore_TS.patch, open-sourc=
e.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually ver=
y similar. Because of that, it is possible to design better compression tha=
n general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save m=
emory in cache as well as speeding seeks within HFileBlocks. It should impr=
ove performance a lot, if key lengths are larger than value lengths. For ex=
ample, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length =3D ~ 90 bytes , value length =3D =
8 bytes) shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio t=
han LZO). Moreover, it should allow far more efficient seeking which should=
 improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of th=
e savings are due to prefix compression, int128 encoding, timestamp diffs a=
nd bitfields to avoid duplication. That way, comparisons of compressed data=
 can be much faster than a byte comparator (thanks to prefix compression an=
d bitfields).
> In order to implement it in HBase two important changes in design will be=
 needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seekin=
g and iterating; access to uncompressed buffer in HFileBlock will have bad =
performance
> -extend comparators to support comparison assuming that N first bytes are=
 equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=3DRe+prefix+com=
pression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira