hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Kennedy <james.kenn...@troove.net>
Subject HMaster won't die waiting for RegionServer that is already dead
Date Fri, 21 Jan 2011 07:28:27 GMT
I've come across a strange bug that I'm having trouble debugging.
Basically I have a seed application that is executed via maven and runs a single JVM ApplicationStarter
that starts up hdfs, regionserver, hmaster threads. It does some seeding then shuts those
down in reverse order.

So this isn't a typical way of running hbase to be sure. However it has always worked until
I upgraded to HBase 0.90.0.
I didn't notice it when I was originally testing 0.90.0 because it only seems to be happening
on our EC2.small build server node when I run this particular seeder.

Running the same thing locally on my mac works.

Attached is the error output starting from when the HRegionServer.stop() is called to when
HMaster.shutdown() is called and it starts looping forever in letRegionServersShutdown().

It looks like RegionServerTracker is getting to "RegionServer ephemeral node deleted, processing
expiration" but then because it can't get the HServerInfo it doesn't follow-through with actually
expiring it. 

Does anyone have any ideas as to why this might be happening?


James Kennedy
Project Manager
Troove Inc.

View raw message