hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bogdan Ghidireac <ghidir...@gmail.com>
Subject Region Server does not shutdown gracefully
Date Wed, 21 Sep 2011 08:07:10 GMT
Hello,

I have an HBase 0.90.3 fleet with more than 100 hosts distributed
across 3 datacenters. Yesterday, we experienced some network issues
for several minutes and the region servers from one data center lost
the connectivity with the namenode. They started the shutdown sequence
but about 20 hosts were unable to complete it successfully. This is
bad for us because we have to restart them manually.

I checked the logs and it looks like the embedded Jetty does not close
all the threads. Here is a sample from the logs.

[FATAL] (IPC Server handler 5 on 60020)
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
server serverName=pa-hbase-datanode-na-7031,60020,1316498497389,
load=(requests=19902, regions=32, usedHeap=582, maxHeap=1820): File
System not available { java.io.IOException: File system is not
available …..
[INFO] (IPC Server handler 5 on 60020)
org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020:
exiting
[WARN] (regionserver60020) org.mortbay.log: 1 threads could not be stopped
[INFO] (regionserver60020)
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
exiting

This is a sample from the stack trace dump on that host. There is one
Jetty thread in runnable state.

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):
"1692369893@qtp-1688716382-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aaab064e000
nid=0x2841 runnable [0x0000000042cf8000]
   java.lang.Thread.State: RUNNABLE
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:724)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Is my assumption correct? Should I open a JIRA?


Regards,
Bogdan

Mime
View raw message