Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 27 Oct 2016 13:12:58 +0000 (UTC)
From: "Duo Zhang (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13013983.1476985746000.101112.1477573978480@Atlassian.JIRA>
In-Reply-To: <JIRA.13013983.1476985746000@Atlassian.JIRA>
References: <JIRA.13013983.1476985746000@Atlassian.JIRA> <JIRA.13013983.1476985746377@arcas>
Subject: [jira] [Commented] (HBASE-16891) Try copying to the Netty ByteBuf
 directly from the WALEdit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 27 Oct 2016 13:13:00 -0000


    [ https://issues.apache.org/jira/browse/HBASE-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611828#comment-15611828 ] 

Duo Zhang commented on HBASE-16891:
-----------------------------------

I do not think this is a good idea. You should follow the semantic of OutputStream as the you just use OutputStream as the CellEncoder's parameter, not a subclass of OutputStream. I think most implementations of OutputStream will only write a byte, not an int...

> Try copying to the Netty ByteBuf directly from the WALEdit
> ----------------------------------------------------------
>
>                 Key: HBASE-16891
>                 URL: https://issues.apache.org/jira/browse/HBASE-16891
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>    Affects Versions: 2.0.0
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>
>         Attachments: HBASE-16891.patch
>
>
> -> The FanOutOneBlockAsyncDFSOutput is a much sophisticated dfs client model that works with Netty ByteBuf. Here we hold on connection to the datanodes using
> Netty Channels. And the idea is to write data direclty to these channels.
> AsyncHLog gets an append call. The AysncWAL uses the HBase's ByteArrayOutputSTream and so the content of the cell is written to this BAOS and that is again
> copied to the netty Bytebuf in the FanOutOneBlockAsyncDFSOutput.
> So when the sync call happens this FanoutDFSoutput does the checksum calcualtion itself and then writes the content of this buffer direclty to the DN channel.
> -> In case of FSHLOg this is different. When an append call comes we direclty write the content to the FSDataOutputStream (it is copied to this stream).
> Then here internally there is a checkSum calculation that happens. when a sync call happens there is noth ing to do except to notify the NN to flush the latest
> data.
> AS we can see from the above that there are two copies in AsyncWAL
> -> From the Cell to the BAOS 
> -> From the BAOS to the Netty byte buf
> -> On sync() call, do check sum and finally flush the netty byte buf to the DN channel
> In case of FSHLog
> -> From cell to the FSDataoutputstream. data is copied. Check sum happens here.
> -> Sync call just tries to notify the NN.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)