hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19344) improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput
Date Sat, 25 Nov 2017 13:44:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265720#comment-16265720
] 

Duo Zhang commented on HBASE-19344:
-----------------------------------

See the comments of syncFailed method in AsyncFSWAL

{code}
  private void syncFailed(Throwable error) {
    LOG.warn("sync failed", error);
    // Here we depends on the implementation of FanOutOneBlockAsyncDFSOutput and netty.
    // When error occur, FanOutOneBlockAsyncDFSOutput will fail all pending flush requests.
It
    // is execute inside EventLoop. And in DefaultPromise in netty, it will notifyListener
    // directly if it is already in the EventLoop thread. And in the listener method, it will
    // call us. So here we know that all failed flush request will call us continuously, and
    // before the last one finish, no other task can be executed in EventLoop. So here we
are
    // safe to use writerBroken as a guard.
    // Do not forget to revisit this if we change the implementation of
    // FanOutOneBlockAsyncDFSOutput!
    consumeLock.lock();
    try {
      if (writerBroken) {
        return;
      }
      writerBroken = true;
      if (waitingRoll) {
        readyForRolling = true;
        readyForRollingCond.signalAll();
      }
    } finally {
      consumeLock.unlock();
    }
    for (Iterator<FSWALEntry> iter = unackedAppends.descendingIterator(); iter.hasNext();)
{
      toWriteAppends.addFirst(iter.next());
    }
    highestUnsyncedTxid = highestSyncedTxid.get();
    // request a roll.
    requestLogRoll();
  }
{code}

It is not straight-forward to use different event loop for AsyncDFSOutput. But anyway, I think
this is a way to increase performance as we can shard the computations from one thread to
several threads.

> improve asyncWAL by using Independent thread for netty #IO in FanOutOneBlockAsyncDFSOutput

> -------------------------------------------------------------------------------------------
>
>                 Key: HBASE-19344
>                 URL: https://issues.apache.org/jira/browse/HBASE-19344
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>    Affects Versions: 2.0.0-beta-1
>            Reporter: Chance Li
>            Assignee: Chance Li
>             Fix For: 2.0.0
>
>         Attachments: HBASE-19344-branch2.patch, wal-1-test-result.png, wal-8-test-result.png
>
>
> The logic now is that the netty #IO thread and asyncWal's thread are the same one.
> Improvement proposal:
> 1, Split into two.
> 2, All multiWal share the netty #IO thread pool. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message