Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Message-ID: <30859941.13291275947418192.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:50:18 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876436#action_12876436 ] 

Luke Lu commented on HADOOP-5793:
---------------------------------

It's written from scratch based on the original BM paper to scratch an itch, which turned out to be stable and fast enough. Zvents owned the copyright for this particular version, however, and I don't know the status of the transfer of the copyright to Hypertable inc. OTOH, a shorter and faster version can be written from scratch again if my current employer (Yahoo) is feeling even more generous than she already is :)

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.