hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hamed Madani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8782) Thrift2 can not parse values when using framed transport
Date Mon, 24 Jun 2013 18:29:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692214#comment-13692214

Hamed Madani commented on HBASE-8782:

Well from what I understand ByteBuffer.array() returns the internal array of the buffer. But
getBytes() return a new subset of internal array. getBytes() extract this small array by looking
at position and limit variables of byteBuffer. With Binary protocol ByteBuffer.readBinary()
returns a new ByteBuffer with a small internal array. (position=0, size= "size of our useful
data). With framed transport, however, ByteBuffer.readBinary() returns the original *trans_*
array, but with new *position* and *limit* variables. so internal arrays with framed transport
are very large containing all the data in one connection. 

As for solution, my first solution to avoid copying the array was to modify HtableInterface
to accept ByteBuffer as input and separately take care of other cases in checkAndPut() and
checkAndDelete(). However, I can see that means adding to HTableInterface!

I found *org.apache.thrift.TBaseHelper.byteBufferToByteArray* to be a more efficient function
for this use case that Bytes.getBytes(). 

  public static byte[] byteBufferToByteArray(ByteBuffer byteBuffer) {
    if (wrapsFullArray(byteBuffer)) {
      return byteBuffer.array();
    byte[] target = new byte[byteBuffer.remaining()];
    byteBufferToByteArray(byteBuffer, target, 0);
    return target;

  public static boolean wrapsFullArray(ByteBuffer byteBuffer) {
    return byteBuffer.hasArray()
      && byteBuffer.position() == 0
      && byteBuffer.arrayOffset() == 0
      && byteBuffer.remaining() == byteBuffer.capacity();

  public static int byteBufferToByteArray(ByteBuffer byteBuffer, byte[] target, int offset)
    int remaining = byteBuffer.remaining();
        byteBuffer.arrayOffset() + byteBuffer.position(),
    return remaining;


the above function is more efficient because for binary protocol it simply returns the inner
array with .array() and for framed protocol it uses system.arraycopy rather than a for loop
to copy the elements. Also above function avoids byteBuffer.duplicate().

If you also think this is a better alternative than getBytes() I can make a new patch using
byteBufferToByteArray() instead of getBytes();
> Thrift2 can not parse values when using framed transport
> --------------------------------------------------------
>                 Key: HBASE-8782
>                 URL: https://issues.apache.org/jira/browse/HBASE-8782
>             Project: HBase
>          Issue Type: Bug
>          Components: Thrift
>    Affects Versions: 0.95.1
>            Reporter: Hamed Madani
>         Attachments: HBASE_8782.patch
> ThriftHBaseServiceHandler.java use .array() on table names , and values (family , qualifier
in checkandDelete , etc) which resulted in incorrect values with framed transport. Replacing
.array() with getBytes() fixed this problem. I've attached the patch
> EDIT: updated the patch to cover checkAndPut(), checkAndDelete()

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message