hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Can master detect sudden region server death?
Date Sat, 14 Aug 2010 00:04:01 GMT
On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <james.kennedy@troove.net> wrote:
> We've recently updated the hbase-transactional-tableindexed extension to work with the
latest 0.89.20100726 version of HBase (still to be pushed).

Good.  Please update the list when you push out the change.

> All HBase tests are passing but then when we started to write our own and test true sudden
HRegionServer death we ran into trouble.
> It seems that the HMaster does not recognize the kill even after many minutes.  Client
requests are blocked and the log continues to repeat the logs below.
> We realized that HBase's own tests that require RegionServer death use abort() and not
kill() which does enough cleanup to inadequately simulate a sudden (e.g. JVM crash) death.
> As an experiment I made HRegionServer.kill() public and modified HBaseMiniCluster to
call that from abort() instead.  Now a test like TestMasterTransitions will exhibit similar
behaviour:  The HMaster never notices the RegionServer is gone.

Odd.  I added kill.  Its used in TestMasterTransistions currently.

Are you sure the master doesn't notice the server gone?   Maybe the RS
is dead in all but the ZK Client session thread so its lease is not
timing out against the ZK ensemble?

The other issue that can get in the way of a RS recovery is replay of
logs; the master needs to grab the lease on the WAL log that the dead
regionserver was writing.  This can take a little while.  It won't
work at all if you are not using an hdfs that doesn't support
'append'; i.e. the hadoop that is in hbase lib dir or build your own
out of apache hadoop branch-0.20-append branch (or get cdh3b2).  Is
this the issue (Tail the log and look for failure to obtain file

An incomplete server crash recovery will show the exceptions you pasted below.

> Could it really be that sudden region server death is not handled in hbase?
> Or more likely is this a failure of the testing framework to adequately simulate kill

Its kinda hard to simulate kill -9 in unit tests.  The 'kill' method
above was a pale attempt at it in our unit testing context.  I've
manually kill -9'd nodes -- RS and DN -- to simulate node death
testing 0.89.x builds and all seems to be basically working.

On making it so master notices downed servers the faster so it'll
start up recovery the sooner, thats a case of playing with the zk
session timeout but coming in the other direction is tuning GC so GC
pause is not > ZK session expiration.

You also want to ensure that there is not too much of a backlog of WAL
logs else recovery which involves splitting all WALs that had
outstanding edits on the crashed server has less work to do when it
runs.  Default is maximum backlog of 32 WALs.  You might want to tune
this down though there is a currently an issue because it seems like
we can overrun this limit.

Hope this helps James,

> James Kennedy
> Project Manager
> Troove Inc.
> -------------------------------
> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO  oop.hbase.master.ServerManager  -
2 region servers, 0 dead, average load 3.5
> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO  adoop.hbase.master.BaseScanner  -
RegionManager.metaScanner scanning meta region {server:, regionname: .META.,,1.1028785192,
startKey: <>}
> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN  adoop.hbase.master.BaseScanner  -
Scan one META region: {server:, regionname: .META.,,1.1028785192, startKey:
> java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
>        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
>        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
>        at $Proxy10.openScanner(Unknown Source)
>        at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
>        at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>        at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>        at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
>        at org.apache.hadoop.hbase.Chore.run(Chore.java:68)

View raw message