hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
Date Sat, 25 Nov 2017 13:44:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265720#comment-16265720

Duo Zhang commented on HBASE-19344:

See the comments of syncFailed method in AsyncFSWAL

  private void syncFailed(Throwable error) {
    LOG.warn("sync failed", error);
    // Here we depends on the implementation of FanOutOneBlockAsyncDFSOutput and netty.
    // When error occur, FanOutOneBlockAsyncDFSOutput will fail all pending flush requests.
    // is execute inside EventLoop. And in DefaultPromise in netty, it will notifyListener
    // directly if it is already in the EventLoop thread. And in the listener method, it will
    // call us. So here we know that all failed flush request will call us continuously, and
    // before the last one finish, no other task can be executed in EventLoop. So here we
    // safe to use writerBroken as a guard.
    // Do not forget to revisit this if we change the implementation of
    // FanOutOneBlockAsyncDFSOutput!
    try {
      if (writerBroken) {
      writerBroken = true;
      if (waitingRoll) {
        readyForRolling = true;
    } finally {
    for (Iterator<FSWALEntry> iter = unackedAppends.descendingIterator(); iter.hasNext();)
    highestUnsyncedTxid = highestSyncedTxid.get();
    // request a roll.

It is not straight-forward to use different event loop for AsyncDFSOutput. But anyway, I think
this is a way to increase performance as we can shard the computations from one thread to
several threads.

> improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput

> -------------------------------------------------------------------------------------------
>                 Key: HBASE-19344
>                 URL: https://issues.apache.org/jira/browse/HBASE-19344
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>    Affects Versions: 2.0.0-beta-1
>            Reporter: Chance Li
>            Assignee: Chance Li
>             Fix For: 2.0.0
>         Attachments: HBASE-19344-branch2.patch, wal-1-test-result.png, wal-8-test-result.png
> The logic now is that the netty #IO thread and asyncWal's thread are the same one.
> Improvement proposal:
> 1, Split into two.
> 2, All multiWal share the netty #IO thread pool. 

This message was sent by Atlassian JIRA

View raw message