hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7728) deadlock occurs between hlog roller and hlog syncer
Date Fri, 01 Feb 2013 23:52:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569272#comment-13569272
] 

stack commented on HBASE-7728:
------------------------------

Looks like I just ran into this:

{code}
Java stack information for the threads listed above:
===================================================
"IPC Server handler 49 on 10304":
        at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1121)
        - waiting to lock <0x0000000583d75420> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.appendNoSync(HLog.java:1167)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2212)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1960)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3454)
        at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
"regionserver10304.logRoller":
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1302)
        - waiting to lock <0x0000000583d75438> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1279)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1432)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:865)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:646)
        - locked <0x0000000583d75420> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
        at java.lang.Thread.run(Thread.java:662)
"IPC Server handler 46 on 10304":
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1312)
        - waiting to lock <0x0000000583d75420> (a java.lang.Object)
        - locked <0x0000000583d75438> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1436)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5176)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2232)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1960)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3454)
        at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

Found 1 deadlock.
{code}

This is a 0.94.3 or so.
                
> deadlock occurs between hlog roller and hlog syncer
> ---------------------------------------------------
>
>                 Key: HBASE-7728
>                 URL: https://issues.apache.org/jira/browse/HBASE-7728
>             Project: HBase
>          Issue Type: Bug
>          Components: wal
>    Affects Versions: 0.94.2
>         Environment: Linux 2.6.18-164.el5 x86_64 GNU/Linux
>            Reporter: Wang Qiang
>            Assignee: Ted Yu
>            Priority: Blocker
>             Fix For: 0.96.0, 0.94.5
>
>         Attachments: 7728-0.94-simplified.txt, 7728-0.94.txt, 7728-0.94-v2.txt, 7728-suggest-0.96.txt,
7728-suggest.txt, 7728-v1.txt, 7728-v2.txt, 7728-v3.txt, 7728-v4.txt
>
>
> the hlog roller thread and hlog syncer thread may occur dead lock with the 'flushLock'
and 'updateLock', and then cause all 'IPC Server handler' thread blocked on hlog append. the
jstack info is as follow :
> "regionserver60020.logRoller":
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1305)
>         - waiting to lock <0x000000067bf88d58> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:876)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:657)
>         - locked <0x000000067d54ace0> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
>         at java.lang.Thread.run(Thread.java:662)
> "regionserver60020.logSyncer":
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1314)
>         - waiting to lock <0x000000067d54ace0> (a java.lang.Object)
>         - locked <0x000000067bf88d58> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1235)
>         at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message