hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16891) Try copying to the Netty ByteBuf directly from the WALEdit
Date Fri, 28 Oct 2016 08:53:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15614811#comment-15614811

Anoop Sam John commented on HBASE-16891:

U can see in EnsureKvEncoder
public void write(Cell cell) throws IOException {
      // Make sure to write tags into WAL
      ByteBufferUtils.putInt(this.out, KeyValueUtil.getSerializedSize(cell, true));
      KeyValueUtil.oswrite(cell, this.out, true);
public static int oswrite(final Cell cell, final OutputStream out, final boolean withTags)
      throws IOException {
    if (cell instanceof ExtendedCell) {
      return ((ExtendedCell)cell).write(out, withTags);
    } else {
All our cells in server side will be ExtendedCell type.
Pls see OffheapKeyValue
public int write(OutputStream out, boolean withTags) throws IOException {
    int length = getSerializedSize(withTags);
    ByteBufferUtils.copyBufferToStream(out, this.buf, this.offset, length);
    return length;
And then BBUtil
public static void copyBufferToStream(OutputStream out, ByteBuffer in,
      int offset, int length) throws IOException {
    if (out instanceof ByteBufferSupportOutputStream) {
      ((ByteBufferSupportOutputStream) out).write(in, offset, length);
    } else if (in.hasArray()) {
      out.write(in.array(), in.arrayOffset() + offset, length);
    } else {
      for (int i = 0; i < length; ++i) {
        out.write(toByte(in, offset + i));
So ya in Encoders we have OutputSTream type only.. Here in this util there is a type check
and we avoid on heap copy from DBB.

> Try copying to the Netty ByteBuf directly from the WALEdit
> ----------------------------------------------------------
>                 Key: HBASE-16891
>                 URL: https://issues.apache.org/jira/browse/HBASE-16891
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>    Affects Versions: 2.0.0
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>         Attachments: HBASE-16891-v1.patch, HBASE-16891.patch
> -> The FanOutOneBlockAsyncDFSOutput is a much sophisticated dfs client model that
works with Netty ByteBuf. Here we hold on connection to the datanodes using
> Netty Channels. And the idea is to write data direclty to these channels.
> AsyncHLog gets an append call. The AysncWAL uses the HBase's ByteArrayOutputSTream and
so the content of the cell is written to this BAOS and that is again
> copied to the netty Bytebuf in the FanOutOneBlockAsyncDFSOutput.
> So when the sync call happens this FanoutDFSoutput does the checksum calcualtion itself
and then writes the content of this buffer direclty to the DN channel.
> -> In case of FSHLOg this is different. When an append call comes we direclty write
the content to the FSDataOutputStream (it is copied to this stream).
> Then here internally there is a checkSum calculation that happens. when a sync call happens
there is noth ing to do except to notify the NN to flush the latest
> data.
> AS we can see from the above that there are two copies in AsyncWAL
> -> From the Cell to the BAOS 
> -> From the BAOS to the Netty byte buf
> -> On sync() call, do check sum and finally flush the netty byte buf to the DN channel
> In case of FSHLog
> -> From cell to the FSDataoutputstream. data is copied. Check sum happens here.
> -> Sync call just tries to notify the NN.

This message was sent by Atlassian JIRA

View raw message