Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1564849389.1218520726047.JavaMail.jira@brutus>
Date: Mon, 11 Aug 2008 22:58:46 -0700 (PDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-818) Deadlock running 'flushSomeRegions'
In-Reply-To: <2064869935.1218498884306.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621704#action_12621704 ] 

stack commented on HBASE-818:
-----------------------------

Suggest moving the line 'this.flushListener.request(this);' outside of the synchronize on WriteState.  Testing here, it seems to do the trick.  There's no need of the flush request being inside the WriteState synchronize block since the scheduling of flushes is async anyways.

> Deadlock running 'flushSomeRegions'
> -----------------------------------
>
>                 Key: HBASE-818
>                 URL: https://issues.apache.org/jira/browse/HBASE-818
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>
> Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 19 on 60020":
>   waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
>   which is held by "IPC Server handler 16 on 60020"
> "IPC Server handler 16 on 60020":
>   waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
>   which is held by "IPC Server handler 2 on 60020"
> "IPC Server handler 2 on 60020":
>   waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
>   which is held by "IPC Server handler 16 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 19 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
>         - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 16 on 60020":
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
>         - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
>         - locked <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
>         - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 2 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
>         - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
>         at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
>         - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> Found 1 deadlock.
> {code}
> Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.