Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Date: Thu, 10 Jul 2014 01:55:04 +0000 (UTC)
From: "Jerry He (JIRA)" <jira@apache.org>
To: dev@hbase.apache.org
Message-ID: <JIRA.12726498.1404957227026.12845.1404957304713@arcas>
In-Reply-To: <JIRA.12726498.1404957227026@arcas>
References: <JIRA.12726498.1404957227026@arcas>
Subject: [jira] [Created] (HBASE-11488) cancelTasks in SubprocedurePool can
 hang during task error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Jerry He created HBASE-11488:
--------------------------------

             Summary: cancelTasks in SubprocedurePool can hang during task error
                 Key: HBASE-11488
                 URL: https://issues.apache.org/jira/browse/HBASE-11488
             Project: HBase
          Issue Type: Bug
          Components: snapshots
    Affects Versions: 0.98.3, 0.96.1, 0.99.0
            Reporter: Jerry He
            Assignee: Jerry He
            Priority: Minor


During snapshot on the region server side, if one RegionSnapshotTask throws exception, we will cancel other tasks.
In RegionServerSnapshotManager.SnapshotSubprocedurePool.waitForOutstandingTasks():
{code}
      LOG.debug("Waiting for local region snapshots to finish.");

      int sz = futures.size();
      try {
        // Using the completion service to process the futures that finish first first.
        for (int i = 0; i < sz; i++) {
          Future<Void> f = taskPool.take();
          f.get();
          if (!futures.remove(f)) {
            LOG.warn("unexpected future" + f);
          }
          LOG.debug("Completed " + (i+1) + "/" + sz +  " local region snapshots.");
        }
        LOG.debug("Completed " + sz +  " local region snapshots.");
        return true;
      } catch (InterruptedException e) {
        LOG.warn("Got InterruptedException in SnapshotSubprocedurePool", e);
        if (!stopped) {
          Thread.currentThread().interrupt();
          throw new ForeignException("SnapshotSubprocedurePool", e);
        }
        // we are stopped so we can just exit.
      } catch (ExecutionException e) {
        if (e.getCause() instanceof ForeignException) {
          LOG.warn("Rethrowing ForeignException from SnapshotSubprocedurePool", e);
          throw (ForeignException)e.getCause();
        }
        LOG.warn("Got Exception in SnapshotSubprocedurePool", e);
        throw new ForeignException(name, e.getCause());
      } finally {
        cancelTasks();
      }
{code}
If  f.get() throws ExecutionException (for example, caused by NotServingRegionException), we will call cancelTasks().
In cancelTasks():
{code}
     ...
     // evict remaining tasks and futures from taskPool.
     while (!futures.isEmpty()) {
        // block to remove cancelled futures;
        LOG.warn("Removing cancelled elements from taskPool");
        futures.remove(taskPool.take());
      }
{code}

For example, suppose we have 3 tasks, the first one fails and we get an exception when we do:
{code}
          Future<Void> f = taskPool.take();
          f.get();
{code}
We didn't remove the 'f' from the 'futures' list yet, but we already take one from taskPool.
As a result, there are 3 in 'futures' list, but only 2 remain in taskPool.
We'll block on taskPool.take() in the above cancelTasks() code.

The end result is that the procedure will always fail timeout exception in the end. 
We could have bailed out earlier with the real cause.


--
This message was sent by Atlassian JIRA
(v6.2#6252)