hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2404) native fast compression codec
Date Fri, 02 Apr 2010 17:16:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852870#action_12852870

Jonathan Gray commented on HBASE-2404:

bq. the way HFile uses LZO has functional equivalence to prefix compression on keys

What exactly do you mean by that?  I understand that repeated prefixes will compress well
with lots of codecs, but is there something special about how we use LZO in HFile that makes
it more prefix friendly?  And if we added prefix compression, would that then mean we wouldn't
use LZO?  As I understand it, those things can still be complimentary, and from what I recall
BigTable uses both prefix compression and LZW.

> native fast compression codec
> -----------------------------
>                 Key: HBASE-2404
>                 URL: https://issues.apache.org/jira/browse/HBASE-2404
>             Project: Hadoop HBase
>          Issue Type: New Feature
>            Reporter: Andrew Purtell
> We often recommend enabling LZO on tables, most users see big wins. LZO is roughly comparable
to BigTable LZW, also the way HFile uses LZO has functional equivalence to prefix compression
on keys. However, LZO is GPL licensed, so a series of install steps are required: http://wiki.apache.org/hadoop/UsingLzoCompression
. It's easy to miss a step or get it wrong. If so, all writes on a table (re)configured to
use LZO will fail. 
> Hadoop, well, Java, has native support for gzip compression but it is too slow generally;
is a good option however for archival tables. 
> This issue is about considering bundling or creating a comparable alternate to LZO which
is ASF 2.0 license compatible. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message