Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 15 Jun 2013 20:47:20 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12653044.1371329155193.120741.1371329240213@arcas>
In-Reply-To: <JIRA.12653044.1371329155193@arcas>
References: <JIRA.12653044.1371329155193@arcas>
Subject: [jira] [Created] (HBASE-8747) Log when blocked longer than expected
 on a lock/synchronized block
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

stack created HBASE-8747:
----------------------------

             Summary: Log when blocked longer than expected on a lock/synchronized block
                 Key: HBASE-8747
                 URL: https://issues.apache.org/jira/browse/HBASE-8747
             Project: HBase
          Issue Type: Improvement
          Components: Usability
    Affects Versions: 0.95.2
            Reporter: stack
            Assignee: stack
            Priority: Critical


Recently, on an internal support thread, the support folks asked that HBase surface when it is blocked up internally or an internal resource is highly contended.  I replied in essence that this would be 'too hard'.  Our Todd instead suggested:

{quote}
...straw man proposal:

- swap out all our use of sychronized (...) for ReentrantLock (probably most of our locking is already based on j.u.c locks anyway for rwlock purposes)
- build a wrapper around ReentrantLock that looks something like:

class AwesomeLock implements Lock {
  AwesomeLock(Lock realLock, int expectedAcquisitionTime, String metricName) {
    this.realLock = realLock;
  }
  public void lock() {
    if (realLock.tryLock(expectedAcquisitionTime, MILLIS)) { return; }
    Stopwatch sw = new Stopwatch();
    // collect realLock.getOwner()
    realLock.lock();
    long elapsed = sw.elapsed() + expectedAcquisitionTime;
    if (elapsed > expected * 2) { LOG.warn("Waited too long for lock which was held by owner " + owner); }
    collect metrics about how long the lock took, the lock name etc
  }
}

Basically the idea is that any possible point of contention should emit log messages and metrics when contended.
{quote}

Let me do the above over the next week or so.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira