hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ryan rawson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
Date Thu, 09 Feb 2012 07:14:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204317#comment-13204317

ryan rawson commented on HBASE-5355:

I had used a number of mechanisms to attempt to reduce the size of the reply.  In no particular

- compression (not using the codec stream interface, using a java impl of lzo or some other
fast compression)
- custom-compression, reducing field dup

The bottom line was that the late-compression was pretty expensive to reduce the size of a
reply. Taking 50-100ms to compress more or less negated the entire benefit at a 5mb response.
I ended up dropping this since it wasn't going to work.

A while back I saw someone who had proposed a compressed representation of KV that had 'natural
prefix' compression.  It took advantage of the fact that KVs are typically stored sorted,
so one could have a 'this KV has the same row as the previous' flag, and ditto for columns,

Aside from that, it might make sense to use the prefix compressed data raw in the RPC response
(as from the blocks), that way there is no re-compression penalty.
> Compressed RPC's for HBase
> --------------------------
>                 Key: HBASE-5355
>                 URL: https://issues.apache.org/jira/browse/HBASE-5355
>             Project: HBase
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.89.20100924
>            Reporter: Karthik Ranganathan
>            Assignee: Karthik Ranganathan
> Some application need ability to do large batched writes and reads from a remote MR cluster.
These eventually get bottlenecked on the network. These results are also pretty compressible
> The aim here is to add the ability to do compressed calls to the server on both the send
and receive paths.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message