hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Hu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-22309) Replace the Shipper interface by using ExtendCell#retain or ExtendCell#release
Date Thu, 25 Apr 2019 09:09:00 GMT
Zheng Hu created HBASE-22309:

             Summary: Replace the Shipper interface by using ExtendCell#retain or ExtendCell#release
                 Key: HBASE-22309
                 URL: https://issues.apache.org/jira/browse/HBASE-22309
             Project: HBase
          Issue Type: Bug
            Reporter: Zheng Hu
            Assignee: Zheng Hu

We've some discussion about the Shipper interface. 

 * This interface denotes a scanner as one which can ship cells. Scan operation do many RPC
 * to server and fetch N rows/RPC. These are then shipped to client. At the end of every such
 * {@link #shipped()} will get called.
public interface Shipper {

   * Called after a batch of rows scanned and set to be returned to client. Any in between
   * can be done here.
  void shipped() throws IOException;

it seems not an elegant ways...for example: 
1.   we want to keep the previous cell in the scanner,  we must deep clone the kv before an
shipping, otherwise the ship will free all the ByteBuffers, the prevCell will point to an
unknown area. 
2.   when switch from PREAD to STREAM in a long scan, we also have accomplish this in shipped()
method. if not , once we close the PREAD scanner,  the un-shipped cell will also point to
an unknown memory area, because the scanner closing free all ByteBuffes.

If we change to use refCnt to manage the RPC memory release or retain,  we can just call prevCell.retain..
then it memory won't be free unless the prevCell reach the end of life and call prevCell#release.
 I mean we can replace all the shipper logic by using  cell#release and cell#retain.  

One concern is about the API,  actually the ExtendCell is an pure server side type,  we can
make the ExtendCell extend the Netty's ReferenceCounted interface , and provide an retain()
and release() methods.   we won't maitain an refCnt in ExtendCell, the refCnt is still in
HFileBlock.  Once we encoded an ExtendCell to CellScanner,  we can release the extendCell,
and it will release its blocks backend... so in theory,  will have no performance loss...

Anyway, that would be a big change,  so maybe need to create another new feature branch to
address this. 

This message was sent by Atlassian JIRA

View raw message