hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Hanging regionservers
Date Mon, 19 Jul 2010 16:21:58 GMT
https://issues.apache.org/jira/browse/HBASE-2248 is fixed in hbase 0.20.4
and beyond.
Upgrading to cdh3b2 should fix that issue.

On Mon, Jul 19, 2010 at 8:55 AM, Luke Forehand <
luke.forehand@networkedinsights.com> wrote:

> After looking at the stacktrace on regionserver2 this morning, I seem to be
> experiencing this issue:
>
> https://issues.apache.org/jira/browse/HBASE-2322
>
> Two questions:  Would this issue cause the primary issue of all my region
> servers appearing to hang, and will migrating to cdh3b2 fix this issue?
>
> Thanks
> Luke
>
> On 7/19/10 12:24 AM, "Luke Forehand" <luke.forehand@networkedinsights.com>
> wrote:
>
> Here are pastebin's of my stacktraces and logs.  Note my comment below
> these links.
>
> regionserver 1 stack trace: http://pastebin.com/0n9cDeYh
> regionserver 2 stack trace: http://pastebin.com/8Sppp68h
> regionserver 3 stack trace: http://pastebin.com/qzLEjBN0
>
> regionserver 1 log ~5MB: http://pastebin.com/g3aB5L81
> regionserver 2 log ~5MB: http://pastebin.com/NDEaUbJv
> regionserver 3 log ~5MB: http://pastebin.com/SAVPnr7S
>
> zookeeper 1,2,3 log: http://pastebin.com/33RPTHKX
>
> So...
>
> Am I seeing a deadlock occurring in the regionserver 2 stacktrace?
>
> "IPC Server handler 18 on 60020" - Thread t@65
>   java.lang.Thread.State: WAITING on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@99de7deowned by: IPC Server
handler 17 on 60020
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>        at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:953)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:846)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
>        - locked
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher@4c2fe6bf
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
>        - locked
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher@4c2fe6bf
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1775)
>        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
>   Locked ownable synchronizers:
>        - locked
> java.util.concurrent.locks.ReentrantLock$NonfairSync@5cd62cac
>
>        - locked
> java.util.concurrent.locks.ReentrantLock$NonfairSync@3cf93af4
>
>
> "IPC Server handler 17 on 60020" - Thread t@64
>   java.lang.Thread.State: BLOCKED on java.util.HashSet@1e1b300f owned by:
> regionserver/192.168.200.32:60020.cacheFlusher
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1524)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1509)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1292)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1255)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1781)
>        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
>   Locked ownable synchronizers:
>        - locked
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@99de7de
>
> "regionserver/192.168.200.32:60020.cacheFlusher" - Thread t@18
>   java.lang.Thread.State: WAITING on
> java.util.concurrent.locks.ReentrantLock$NonfairSync@5cd62cac owned by:
> IPC Server handler 18 on 60020
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>        at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>        at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
>        - locked java.util.HashSet@1e1b300f
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
>
>   Locked ownable synchronizers:
>       - None
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message