Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4F159185.80805@zfabrik.de>
Date: Tue, 17 Jan 2012 16:19:33 +0100
From: Henning Blohm <henning.blohm@zfabrik.de>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64;
 rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2
MIME-Version: 1.0
To: user@hbase.apache.org
Subject: ROOT region not assigned
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

After an upgrade of hadoop and hbase (to 0.90.4-cdh3u2) from 0.90 hbase 
and 0.20-append hadoop on a single node test installation everything 
worked fine initially.

Then there was some DNS changes and host name changes which resulted in 
a lot "hostname <oldname> cannot be resolved" problems in the logs and 
the master web interface would only show a stack trace from a bad lookup 
("hostname can't be null").

So I changed the host name back to its old name. All configuration in 
hbase/hadoop points to localhost (i.e. in the *-site.xml, slaves, 
masters, regionservers). We are running distributed mode (but on one 
machine).

Now, the HMaster process does come up again somewhat and I get the web 
interface, but it stays in

Currently running tasks:

Master Startup, Assigning ROOT region, 10050s

i.e. it never continues.

The hadoop data directory was not touched other than during the upgrade 
which completed successfully and data was available after the upgrade. 
No, of course no data is available anymore.

This is really scary, as we plan to do similar upgrades in production 
environments and I would like to understand what could possibly screw 
things up to badly.

The HMaster log shows things like:


2012-01-17 12:48:39,712 INFO 
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition 
timed out:  -ROOT-,,0.70236052 state=OPEN, ts=1326707373329, 
server=application1,60020,1326707331441
2012-01-17 12:48:39,713 ERROR 
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPEN 
for too long, we don't know where region was opened so can't do anything

The region server log has a lot of these:

2012-01-17 13:03:36,774 DEBUG 
org.apache.hadoop.hbase.regionserver.HRegionServer: 
NotServingRegionException; Region is not online: -ROOT-,,0

Help would be great!!

Thanks,
   Henning