hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path
Date Wed, 13 Nov 2013 09:01:45 GMT

     [ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-5945:

    Attachment: 5945v2.txt

Forward port Todd's old idea of saving copies by passing array of ByteBuffers to Channel rather
than compose response in a buffer first before putting it on the wire.  Revives his BufferChain

Here's some comment on the changes.

M hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java

Added getDelimitedMessageAsByteBuffer(final Message m)
Removed no longer used write method and getDelimitedMessageBytes.

M hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java

Changed bytes sent size from int to long (it is long in implementation, just not in interfaces)

A hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BufferChain.java

Class to carry an array of ByteBuffers and that can write the byte buffers in order
out to a GatheringByteChannel in chunks.

M hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java

Refactored response sending to use BufferChain.  We save on a composition of total
response in memory.  Instead we now pass the parts directly to the socket channel.

Stuff is a bit ugly in the sasl handling because have to compose the message
in memory completely so can do the sasl wrapping; this undoes any savings but
I believe we are still ahead because we removed the serializing of the resulting
sasl array to an output stream backed by a new buffer.

> Reduce buffer copies in IPC server response path
> ------------------------------------------------
>                 Key: HBASE-5945
>                 URL: https://issues.apache.org/jira/browse/HBASE-5945
>             Project: HBase
>          Issue Type: Improvement
>          Components: IPC/RPC
>    Affects Versions: 0.95.2
>            Reporter: Todd Lipcon
>            Assignee: stack
>             Fix For: 0.96.1
>         Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, 5945-in-progress.patch,
5945v2.txt, buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt
> The new PB code is sloppy with buffers and makes several needless copies. This increases
GC time a lot. A few simple changes can cut this back down.

This message was sent by Atlassian JIRA

View raw message