hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Corgan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
Date Sun, 30 Mar 2014 20:18:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954825#comment-13954825
] 

Matt Corgan commented on HBASE-10771:
-------------------------------------

One of the problems with ByteBuffer is that it's difficult to copy multiple bytes from BB
to byte[] in a multi-threaded scenario, for example, when the BlockCache is a bunch of DirectBBs
and we want to have many simultaneous reader threads reading from a hot block.  For some reason,
there's not a method to copy a range of bytes from a random position in the DBB, so you have
to either acquire a lock on the DBB to take ownership of its position/limit fields, or you
have to "slice" it to give the reader a copy with its own position/limit.  Both locking and
slicing would be very expensive when trying to assemble encoded Cells in a tight loop.

One solution might be to have a pool of reader threads (i think we already do), and to give
each thread a pre-sliced BB for each big slab in the allocator.  The reader thread could reuse
its private copy of the slab BB:
{code}
ByteRange dest = some ByteRange we want to copy into;
//the threadLocalBlock is a DirectByteBufferByteRange in this case.
//allocator sets position/limit to the block boundaries in the underlying slab.
//allocator manages thread-local pool of blocks and does ref counting.
ByteRange threadLocalBlock = allocator.acquireReadOnlyThreadLocalBlock(blockCacheKey);
dest.copyFrom(threadLocalBlock, srcOffset, destOffset, length);
allocator.releaseReadOnlyThreadLocalBlock(blockCacheKey);

the array-backed ByteRange would do:
class ArrayByteRange implements ByteRange{
  void copyFrom(ByteRange src, int srcOffset, int destOffset, int length){
    if(src.hasArray()){
      System.arraycopy(src.getArray(), srcOffset, this.array, destOffset, length);
    }else{
      DirectByteBuffer srcBuffer = ((DirectByteBuffer)src).getBuffer();
      int previousPosition = srcBuffer.position();
      int previousLimit = srcBuffer.limit();
      srcBuffer.position(srcOffset);
      srcBuffer.limit(srcOffset + length);
      srcBuffer.get(this.array);//copies the bytes
      srcBuffer.position(previousPosition);
      srcBuffer.limit(previousLimit);
    }
  }
}

while the DirectByteBufferByteRange would do:
class DirectByteBufferByteRange implements ByteRange{
  void copyFrom(ByteRange src, int srcOffset, int destOffset, int length){
    if(src.hasArray()){
      byte[] srcArray = ((ArrayByteBuffer)src).getArray();
      this.buffer.put(srcArray, srcOffset, length);//copies the bytes
    }else{
      DirectByteBuffer srcBuffer = ((DirectByteBuffer)src).getBuffer();
      int previousPosition = srcBuffer.position();
      int previousLimit = srcBuffer.limit();
      srcBuffer.position(srcOffset);
      srcBuffer.limit(srcOffset + length);
      this.buffer.put(srcBuffer);//copies the bytes
      srcBuffer.position(previousPosition);
      srcBuffer.limit(previousLimit);
    }
  }
}
{code}

Need to be careful with offset/limit calculations.  They may be more complex than in that
example.

> Primitive type put/get APIs in ByteRange 
> -----------------------------------------
>
>                 Key: HBASE-10771
>                 URL: https://issues.apache.org/jira/browse/HBASE-10771
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 0.99.0
>
>         Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
>
>
> While doing HBASE-10713 I came across the need to write int/long (and read also) from
a ByteRange.  CellBlocks are backed by ByteRange. So we can add such APIs.
> Also as per HBASE-10750  we return a ByteRange from MSLAB and also discussion under HBASE-10191
suggest we can have BR backed HFileBlocks etc.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message