hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: lost .META. table
Date Wed, 22 Sep 2010 00:43:49 GMT
Thanks Stack - worked like a charm, and I think all my data is there.  So
does the content of -ROOT- get overwritten, because I think it was different
than it should have been?


On Tue, Sep 21, 2010 at 7:25 PM, Stack <stack@duboce.net> wrote:

> Ok.  Thanks.  I don't know why its not assigning .META.  Trying
> starting the cluster with a different hbase.rootdir then move the
> -ROOT- from the new hbase.rootdir in place of the -ROOT- you currently
> have.
>
> St.Ack
>
> On Tue, Sep 21, 2010 at 4:17 PM, Stack <stack@duboce.net> wrote:
> > Please put the first 1k lines after startup up in pastebin.  The below
> > is hard to read.  Its scanning the -ROOT- but there is no mention of
> > .META. being assigned out.
> >
> > St.Ack
> >
> > On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <mcorgan@hotpads.com>
> wrote:
> >> I think this is the relevant section:
> >>
> >> 2010-09-21 15:42:11,986 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode41,60020,1285098132069
> >> 2010-09-21 15:42:11,989 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode38,60020,1285098131987
> >> 2010-09-21 15:42:11,989 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098132069 with data 10.214.7.223:60020
> >> 2010-09-21 15:42:11,996 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098131987 with data 10.209.206.15:60020
> >> 2010-09-21 15:42:13,001 DEBUG
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987,
> load:
> >> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
> >> assign=1, regions to give other servers than this=0, isMetaAssign=true
> >> 2010-09-21 15:42:13,001 DEBUG
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
> >> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
> >> 2010-09-21 15:42:13,001 INFO
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
> >> 2010-09-21 15:42:13,085 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Processing MSG_REPORT_OPEN: -ROOT-,,0 from
> HadoopNode38,60020,1285098131987;
> >> 1 of 1
> >> 2010-09-21 15:42:13,130 WARN
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out
> of
> >> safe mode in ZooKeeper:
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists for /hbase/safe-mode
> >> 2010-09-21 15:42:13,130 INFO
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just
> move
> >> on
> >> 2010-09-21 15:42:13,130 INFO
> org.apache.hadoop.hbase.master.RegionManager:
> >> exiting safe mode
> >> 2010-09-21 15:42:13,132 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
> >> /hbase/root-region-server with 10.209.206.15:60020
> >> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020,
> >> regionname: -ROOT-,,0, startKey: <>}
> >> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> >> 2010-09-21 15:42:13,466 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode40,60020,1285098133450
> >> 2010-09-21 15:42:13,468 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098133450 with data 10.192.66.240:60020
> >> 2010-09-21 15:42:13,828 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode39,60020,1285098133654
> >> 2010-09-21 15:42:13,830 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098133654 with data 10.206.26.95:60020
> >> 2010-09-21 15:43:10,256 INFO
> org.apache.hadoop.hbase.master.ServerManager: 4
> >> region servers, 0 dead, average load 0.25
> >> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020,
> >> regionname: -ROOT-,,0, startKey: <>}
> >> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> >> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All
> >> 0 .META. region(s) scanned
> >> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
> >> client connection, connectString=10.249.107.223:2181sessionTimeout=60000
> >>
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
> >> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
> >> connection to server /10.249.107.223:2181
> >> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> 10.249.107.223:56828remote=/10.249.107.223:2181]
> >> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
> >> connection successful
> >>
> >>
> >> On Tue, Sep 21, 2010 at 6:05 PM, Stack <stack@duboce.net> wrote:
> >>
> >>> Ok.  So .META. is there.  What is in your master log on startup?  Its
> >>> supposed to assign out -ROOT- and .META. (when .META. is assigned
> >>> it'll add row to -ROOT-).  I'd say hold off on any of the trickery
> >>> described in the first mail till we see snippet from your master log.
> >>> Lets figure why not assigning .META. before we do any messing.
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mcorgan@hotpads.com>
> wrote:
> >>> > Thanks for responding Stack.  In a 4 slave cluster (rf=3) cluster we
> >>> > disabled an hdfs/hbase node which also seemed to make a second
> >>> regionserver
> >>> > crash.  I'm not sure exactly what went wrong.
> >>> >
> >>> > There is data in the .META. directory
> >>> >
> >>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
> -lsr
> >>> > /hbase/.META.
> >>> > drwxr-xr-x   - root supergroup          0 2010-09-20 16:44
> >>> > /hbase/.META./1028785192
> >>> > -rw-r--r--   3 root supergroup        932 2010-09-20 14:25
> >>> > /hbase/.META./1028785192/.regioninfo
> >>> > drwxr-xr-x   - root supergroup          0 2010-09-21 15:41
> >>> > /hbase/.META./1028785192/info
> >>> > -rw-r--r--   3 root supergroup       5018 2010-09-21 15:41
> >>> > /hbase/.META./1028785192/info/4694688683869779325
> >>> > -rw-r--r--   3 root supergroup          0 2010-09-21 14:48
> >>> > /hbase/.META./1028785192/info/6832908118284006198
> >>> > -rw-r--r--   3 root supergroup     228130 2010-09-21 14:44
> >>> > /hbase/.META./1028785192/info/7773537252698623585
> >>> > drwxr-xr-x   - root supergroup          0 2010-09-21 14:44
> >>> > /hbase/.META./compaction.dir
> >>> >
> >>> >
> >>> > but it appears the -ROOT- region is empty.  If i scan '-ROOT-' in the
> >>> hbase
> >>> > shell i get:
> >>> >
> >>> > hbase(main):001:0> scan '-ROOT-'
> >>> > ROW                          COLUMN+CELL
> >>> > 0 row(s) in 0.0840 seconds
> >>> >
> >>> >
> >>> > even though it appears -ROOT- exists in hdfs.  Any idea why ROOT
> would
> >>> not
> >>> > get loaded from hdfs after a master restart?
> >>> >
> >>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
> -lsr
> >>> > /hbase/-ROOT-
> >>> > drwxr-xr-x   - root supergroup          0 2010-09-21 14:50
> >>> > /hbase/-ROOT-/70236052
> >>> > -rw-r--r--   3 root supergroup        618 2010-09-21 14:50
> >>> > /hbase/-ROOT-/70236052/.regioninfo
> >>> > drwxr-xr-x   - root supergroup          0 2010-09-21 15:41
> >>> > /hbase/-ROOT-/70236052/info
> >>> > -rw-r--r--   3 root supergroup        489 2010-09-21 15:41
> >>> > /hbase/-ROOT-/70236052/info/308400263036654323
> >>> > -rw-r--r--   3 root supergroup        462 2010-09-21 13:43
> >>> > /hbase/-ROOT-/70236052/info/8250773031047005400
> >>> >
> >>> >
> >>> > I'll just follow your suggestion of generating a new .META. and
> copying
> >>> > over.  Thanks,
> >>> >
> >>> > Matt
> >>> >
> >>> >
> >>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <stack@duboce.net> wrote:
> >>> >
> >>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mcorgan@hotpads.com>
> >>> wrote:
> >>> >> > Is there a way to create a new .META. table if I lost it?
 For
> >>> example,
> >>> >> I'm
> >>> >> > getting this exception everywhere:
> >>> >> >
> >>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> >>> (NativeException)
> >>> >> >
> >>> >>
> >>> >> How'd you do that?  Thats pretty impressive.
> >>> >>
> >>> >> >
> >>> >> > The add_table script doesn't appear to handle .META., giving
this
> >>> error:
> >>> >> >
> >>> >> > java.lang.IllegalArgumentException: Illegal first character
<46>
> at 0.
> >>> >> > User-space table names can only start with 'word characters':
i.e.
> >>> >> > [a-zA-Z_0-9]: .META.
> >>> >> >
> >>> >> >
> >>> >> > And the hbase shell will only create user-space table names.
> >>>  Restarting
> >>> >> the
> >>> >> > hbase master also does not appear to re-create it.
> >>> >> >
> >>> >> > I'm using 0.20.6.
> >>> >> >
> >>> >> Whats the master log look like when you start up hbase?  Whats
it
> say
> >>> >> about missing .META.?
> >>> >>
> >>> >> If you do a
> >>> >>
> >>> >> $ bin/hadoop fs -lsr /hbase/.META.
> >>> >>
> >>> >> ... is there anything in your filesystem?
> >>> >>
> >>> >> If nothing there, one thing you could do is run an hbase elsewhere.
> >>> >> Let it create a new "/hbase" dir and then copy the .META. table
from
> >>> >> here up to the filesystem with data (Run add_table.rb after .META.
> is
> >>> >> in place to repopulate its rows).
> >>> >>
> >>> >> St.Ack
> >>> >>
> >>> >
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message