hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
Date Fri, 12 Oct 2012 04:43:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474786#comment-13474786
] 

Lars Hofhansl commented on HBASE-5355:
--------------------------------------

Before we commit this or the trunk patch I'd love to see some numbers comparing this full
compression stream approach with just avoiding duplicate data while serializing from/to the
RegionServer. On both sides we'd have to reassemble the full KVs (unless we finally make a
KV interface), but we can that efficiently if we keep track size of the omitted parts of the
KV and preallocate the space and copy the data in that. That way we'd have the same amount
memory copying (ignoring DMA from the network card for the moment) and can safe bytes on the
wire.
I raised this on the mailing this a while ago, and Andy commented on that somewhere as well.
KV are sorted when traveling over the wire (as a set of Puts/Deletes or in a Result) we can
simple avoid copying the prefix multiple times.
                
> Compressed RPC's for HBase
> --------------------------
>
>                 Key: HBASE-5355
>                 URL: https://issues.apache.org/jira/browse/HBASE-5355
>             Project: HBase
>          Issue Type: Improvement
>          Components: IPC/RPC
>    Affects Versions: 0.89.20100924
>            Reporter: Karthik Ranganathan
>            Assignee: Karthik Ranganathan
>         Attachments: HBASE-5355-0.94.patch
>
>
> Some application need ability to do large batched writes and reads from a remote MR cluster.
These eventually get bottlenecked on the network. These results are also pretty compressible
sometimes.
> The aim here is to add the ability to do compressed calls to the server on both the send
and receive paths.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message