hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4367) Deadlock in MemStore flusher due to JDK internally synchronizing on current thread
Date Sat, 10 Sep 2011 09:25:09 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102020#comment-13102020
] 

Ted Yu commented on HBASE-4367:
-------------------------------

According to the line number above, CompactionRequest.toString() is from this line:
{code}
        LOG.trace("Inserting region in queue. " + newRequest);
{code}
I think the above code was for debugging purpose. Meaning, we can change what CompactionRequest.toString()
returns.

The Date field in CompactionRequest is used in its compareTo() method.
By examining the code, the following is always executed (starting line 61):
{code}
      if (d == null) {
        d = new Date();
      }
{code}
I think we can replace Date field with a long field which carries timestamp.
This would satisfy its usage in compareTo() method and avoid Date.toString() altogether.

Option 2 is to omit the date component in CompactionRequest.toString().

If any of the above options is chosen, please allow me to provide a patch.

> Deadlock in MemStore flusher due to JDK internally synchronizing on current thread
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-4367
>                 URL: https://issues.apache.org/jira/browse/HBASE-4367
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.4
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.92.0
>
>
> We observed a deadlock in production between the following threads:
> - IPC handler thread holding the monitor lock on MemStoreFlusher inside reclaimMemStoreMemory,
waiting to obtain MemStoreFlusher.lock (the reentrant lock member)
> - cacheFlusher thread inside flushRegion holds MemStoreFlusher.lock, and then calls PriorityCompactionQueue.add,
which calls PriorityCompactionQueue.addToRegionsInQueue, which calls CompactionRequest.toString(),
which calls Date.toString. If this occurs just after a GC under memory pressure, Date.toString
needs to reload locale information (stored in a soft reference), so it calls ResourceBundle.loadBundle,
which uses Thread.currentThread() as a synchronizer (see sun bug http://bugs.sun.com/view_bug.do?bug_id=6915621).
Since the current thread is the MemStoreFlusher itself, we have a lock order inversion and
a deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message