Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 13 Nov 2015 07:03:11 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12911749.1447124202000.59205.1447398191078@Atlassian.JIRA>
In-Reply-To: <JIRA.12911749.1447124202000@Atlassian.JIRA>
References: <JIRA.12911749.1447124202000@Atlassian.JIRA>
 <JIRA.12911749.1447124202700@arcas>
Subject: [jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream
 for logging WAL only
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003636#comment-15003636 ] 

stack commented on HBASE-14790:
-------------------------------

[~busbey]

bq. That's old code I've yet to see benchmark justification for.

Here is where having multiple syncers is first challenged: https://issues.apache.org/jira/browse/HBASE-8755?focusedCommentId=13830604&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13830604 It is followed by numbers that show that throughput is better with 5. I remember playing with syncers after ringbuffer went in and arrived again at 5 syncers as best for throughput.

I'll be happy to see them go. Chatting w/ [~Apache9], having a single thread start the syncs with callbacks to take care of letting blocked handlers go sounds cleaner. I'm not sure how it will look just yet. We'll want to keep our group commits fat and we'll want to minimize inter-thread communication/blocking.

[~wheat9]
bq. A less risky route is to combine the effort with the HTTP/2 initiatives of HDFS which allows full control on both the client and the server side. Thoughts?

I'm with [~Apache9]. Lets not broaden the scope of an already involved project by mixing in http/2. Lets also get a success here in hbase first -- we have a very particular need, we have tooling and mechanisms to verify perf and correct behavior -- using a subset of the write API in an async way.... and then move to the general case with this feather in our hat (Einstein published the special theory of relativity before he did the general).


> Implement a new DFSOutputStream for logging WAL only
> ----------------------------------------------------
>
>                 Key: HBASE-14790
>                 URL: https://issues.apache.org/jira/browse/HBASE-14790
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all purposes. But in fact, we do not need most of the features if we only want to log WAL. For example, we do not need pipeline recovery since we could just close the old logger and open a new one. And also, we do not need to write multiple blocks since we could also open a new logger if the old file is too large.
> And the most important thing is that, it is hard to handle all the corner cases to avoid data loss or data inconsistency(such as HBASE-14004) when using original DFSOutputStream due to its complicated logic. And the complicated logic also force us to use some magical tricks to increase performance. For example, we need to use multiple threads to call {{hflush}} when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when logging WAL. For correctness, and also for performance.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)