Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 74.125.82.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=LKH+k2Ss8geR9aBo659Y+Jx2zrBWqWSOgBxZFs3eY24THKVMDyYtsqX0dti7dGSy2x
         V3LmY0icivcy4Y/KMCFWpLs7ysn0Q01/CGPBXtL6FISdIAtzndsMIzC3RE9qSAp8CclA
         u38VPcEJswHBzlvuWaKWqsnHw1UbAba8g8vbE=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <AANLkTinBVvqhLQ3YL8Jv4LS+mU9dy7w9cqu14ExMSpNX@mail.gmail.com>
References: <AANLkTim+mtwk=rHdqBZjqWF5Y3nYO3xtbrhH_okDOCgv@mail.gmail.com>
	<AANLkTinm1KFMjkYVrq9o8qFRvG_yd452BiT8iVttZPL3@mail.gmail.com>
	<AANLkTi=BPjCsOB49A42xsLnhnam1tVK4PQq-aaDc33Nn@mail.gmail.com>
	<AANLkTiksAi96n51fzSjbBJ-OxviC1b+jNq=-CXaS=QEC@mail.gmail.com>
	<AANLkTikpR=Az5tAareHGnJYHgSYgzW203e_4cd6MWi=W@mail.gmail.com>
	<AANLkTinBVvqhLQ3YL8Jv4LS+mU9dy7w9cqu14ExMSpNX@mail.gmail.com>
Date: Tue, 21 Sep 2010 16:25:37 -0700
Message-ID: <AANLkTik2pB28+H300rx+6zM=n-C+53FwNPttH0FLaVG4@mail.gmail.com>
Subject: Re: lost .META. table
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Ok.  Thanks.  I don't know why its not assigning .META.  Trying
starting the cluster with a different hbase.rootdir then move the
-ROOT- from the new hbase.rootdir in place of the -ROOT- you currently
have.

St.Ack

On Tue, Sep 21, 2010 at 4:17 PM, Stack <stack@duboce.net> wrote:
> Please put the first 1k lines after startup up in pastebin. =A0The below
> is hard to read. =A0Its scanning the -ROOT- but there is no mention of
> .META. being assigned out.
>
> St.Ack
>
> On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <mcorgan@hotpads.com> wrote:
>> I think this is the relevant section:
>>
>> 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManage=
r:
>> Received start message from: HadoopNode41,60020,1285098132069
>> 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManage=
r:
>> Received start message from: HadoopNode38,60020,1285098131987
>> 2010-09-21 15:42:11,989 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098132069 with data 10.214.7.223:60020
>> 2010-09-21 15:42:11,996 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098131987 with data 10.209.206.15:60020
>> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManag=
er:
>> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, lo=
ad:
>> (requests=3D0, regions=3D0, usedHeap=3D32, maxHeap=3D5195): total nregio=
ns to
>> assign=3D1, regions to give other servers than this=3D0, isMetaAssign=3D=
true
>> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManag=
er:
>> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
>> (requests=3D0, regions=3D0, usedHeap=3D32, maxHeap=3D5195) 1 regions
>> 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManage=
r:
>> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
>> 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManage=
r:
>> Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131=
987;
>> 1 of 1
>> 2010-09-21 15:42:13,130 WARN
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out=
 of
>> safe mode in ZooKeeper:
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCod=
e =3D
>> NodeExists for /hbase/safe-mode
>> 2010-09-21 15:42:13,130 INFO
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just mo=
ve
>> on
>> 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManage=
r:
>> exiting safe mode
>> 2010-09-21 15:42:13,132 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
>> /hbase/root-region-server with 10.209.206.15:60020
>> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60=
020,
>> regionname: -ROOT-,,0, startKey: <>}
>> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
>> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
>> 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManage=
r:
>> Received start message from: HadoopNode40,60020,1285098133450
>> 2010-09-21 15:42:13,468 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098133450 with data 10.192.66.240:60020
>> 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManage=
r:
>> Received start message from: HadoopNode39,60020,1285098133654
>> 2010-09-21 15:42:13,830 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098133654 with data 10.206.26.95:60020
>> 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManage=
r: 4
>> region servers, 0 dead, average load 0.25
>> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60=
020,
>> regionname: -ROOT-,,0, startKey: <>}
>> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
>> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
>> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner:=
 All
>> 0 .META. region(s) scanned
>> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, connectString=3D10.249.107.223:2181 sessionTimeout=3D=
60000
>> watcher=3Dorg.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatc=
her@72898540
>> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server /10.249.107.223:2181
>> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
>> connection to java.nio.channels.SocketChannel[connected local=3D/
>> 10.249.107.223:56828remote=3D/10.249.107.223:2181]
>> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
>> connection successful
>>
>>
>> On Tue, Sep 21, 2010 at 6:05 PM, Stack <stack@duboce.net> wrote:
>>
>>> Ok. =A0So .META. is there. =A0What is in your master log on startup? =
=A0Its
>>> supposed to assign out -ROOT- and .META. (when .META. is assigned
>>> it'll add row to -ROOT-). =A0I'd say hold off on any of the trickery
>>> described in the first mail till we see snippet from your master log.
>>> Lets figure why not assigning .META. before we do any messing.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mcorgan@hotpads.com> wrot=
e:
>>> > Thanks for responding Stack. =A0In a 4 slave cluster (rf=3D3) cluster=
 we
>>> > disabled an hdfs/hbase node which also seemed to make a second
>>> regionserver
>>> > crash. =A0I'm not sure exactly what went wrong.
>>> >
>>> > There is data in the .META. directory
>>> >
>>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -=
lsr
>>> > /hbase/.META.
>>> > drwxr-xr-x =A0 - root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-20 16:4=
4
>>> > /hbase/.META./1028785192
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 =A0932 2010-09-20 14:25
>>> > /hbase/.META./1028785192/.regioninfo
>>> > drwxr-xr-x =A0 - root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-21 15:4=
1
>>> > /hbase/.META./1028785192/info
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 5018 2010-09-21 15:41
>>> > /hbase/.META./1028785192/info/4694688683869779325
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-21 14:4=
8
>>> > /hbase/.META./1028785192/info/6832908118284006198
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 228130 2010-09-21 14:44
>>> > /hbase/.META./1028785192/info/7773537252698623585
>>> > drwxr-xr-x =A0 - root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-21 14:4=
4
>>> > /hbase/.META./compaction.dir
>>> >
>>> >
>>> > but it appears the -ROOT- region is empty. =A0If i scan '-ROOT-' in t=
he
>>> hbase
>>> > shell i get:
>>> >
>>> > hbase(main):001:0> scan '-ROOT-'
>>> > ROW =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0COLUMN+CELL
>>> > 0 row(s) in 0.0840 seconds
>>> >
>>> >
>>> > even though it appears -ROOT- exists in hdfs. =A0Any idea why ROOT wo=
uld
>>> not
>>> > get loaded from hdfs after a master restart?
>>> >
>>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -=
lsr
>>> > /hbase/-ROOT-
>>> > drwxr-xr-x =A0 - root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-21 14:5=
0
>>> > /hbase/-ROOT-/70236052
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 =A0618 2010-09-21 14:50
>>> > /hbase/-ROOT-/70236052/.regioninfo
>>> > drwxr-xr-x =A0 - root supergroup =A0 =A0 =A0 =A0 =A00 2010-09-21 15:4=
1
>>> > /hbase/-ROOT-/70236052/info
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 =A0489 2010-09-21 15:41
>>> > /hbase/-ROOT-/70236052/info/308400263036654323
>>> > -rw-r--r-- =A0 3 root supergroup =A0 =A0 =A0 =A0462 2010-09-21 13:43
>>> > /hbase/-ROOT-/70236052/info/8250773031047005400
>>> >
>>> >
>>> > I'll just follow your suggestion of generating a new .META. and copyi=
ng
>>> > over. =A0Thanks,
>>> >
>>> > Matt
>>> >
>>> >
>>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <stack@duboce.net> wrote:
>>> >
>>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mcorgan@hotpads.com>
>>> wrote:
>>> >> > Is there a way to create a new .META. table if I lost it? =A0For
>>> example,
>>> >> I'm
>>> >> > getting this exception everywhere:
>>> >> >
>>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>>> (NativeException)
>>> >> >
>>> >>
>>> >> How'd you do that? =A0Thats pretty impressive.
>>> >>
>>> >> >
>>> >> > The add_table script doesn't appear to handle .META., giving this
>>> error:
>>> >> >
>>> >> > java.lang.IllegalArgumentException: Illegal first character <46> a=
t 0.
>>> >> > User-space table names can only start with 'word characters': i.e.
>>> >> > [a-zA-Z_0-9]: .META.
>>> >> >
>>> >> >
>>> >> > And the hbase shell will only create user-space table names.
>>> =A0Restarting
>>> >> the
>>> >> > hbase master also does not appear to re-create it.
>>> >> >
>>> >> > I'm using 0.20.6.
>>> >> >
>>> >> Whats the master log look like when you start up hbase? =A0Whats it =
say
>>> >> about missing .META.?
>>> >>
>>> >> If you do a
>>> >>
>>> >> $ bin/hadoop fs -lsr /hbase/.META.
>>> >>
>>> >> ... is there anything in your filesystem?
>>> >>
>>> >> If nothing there, one thing you could do is run an hbase elsewhere.
>>> >> Let it create a new "/hbase" dir and then copy the .META. table from
>>> >> here up to the filesystem with data (Run add_table.rb after .META. i=
s
>>> >> in place to repopulate its rows).
>>> >>
>>> >> St.Ack
>>> >>
>>> >
>>>
>>
>