hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ludvig Omholt <lu...@wiral.se>
Subject All tables lost after reboot of one region server
Date Fri, 27 Feb 2009 14:11:11 GMT
Hi!

We are running a small hadoop/hbase cluster on two machines, master and
slave. Versions are 0.19. Both machines are running hadoop datanodes and
hbase region servers. The master machine is running the hbase master
node and the hadoop name node. We have seven tables in hbase.

Now, to see how fail safe this setup is I tried rebooting the slave
machine. After the reboot the region server on the slave machine was
started manually by running bin/start-hbase.sh on the master.

It started successfully but when I list tables in the hbase shell all
seven tables are gone.

$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.19.0, r735381, Sun Jan 18 14:29:34 PST 2009
hbase(main):001:0> list
0 row(s) in 0.1380 seconds

This may be relevant parts of the log on the slave region server right
after it has been restarted:

2009-02-27 09:48:02,978 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer: 
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
2009-02-27 09:48:02,979 INFO org.apache.hadoop.ipc.HBaseServer: IPC 
Server handler 6 on 60020: starting
2009-02-27 09:48:02,981 ERROR 
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
	at 
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
	at 
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
	at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)


Files in hadoop seem to be intact. I have tried restarting hadoop and
hbase, but the tables still don't magically reappear as I was hoping
for. What can I do to debug this and hopefully recover my data?

There is a historian directory under /hbase/.META. that contains a data
and an index file, can I try to recover my tables using these somehow?

Thanks for any help,
Ludvig Omholt

Mime
View raw message