hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
Date Thu, 09 Feb 2012 06:07:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204296#comment-13204296
] 

Todd Lipcon commented on HBASE-5355:
------------------------------------

bq. Todd in your proposal above we'd convert KVs to your new form before putting payload on
wire and then undo it on other side?

Yep, something like that. But hopefully it's just reference twiddling and not any actual data
copying.

bq. Long time back Ryan tried to a custom compression before putting stuff on the wire and
then undoing it on other end hoping it would help some w/ latency but he found it upped latency

I remember that too, but if I remember correctly it was going through the whole codec stream
interface, which necessitates extra copies, etc. There's a lot of room for optimization there
by adding new APIs to the hadoop codec stuff that operate on direct byte buffers in-place.
And doing the "application level compression" as described above should be faster as well.
                
> Compressed RPC's for HBase
> --------------------------
>
>                 Key: HBASE-5355
>                 URL: https://issues.apache.org/jira/browse/HBASE-5355
>             Project: HBase
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.89.20100924
>            Reporter: Karthik Ranganathan
>            Assignee: Karthik Ranganathan
>
> Some application need ability to do large batched writes and reads from a remote MR cluster.
These eventually get bottlenecked on the network. These results are also pretty compressible
sometimes.
> The aim here is to add the ability to do compressed calls to the server on both the send
and receive paths.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message