hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane O'Donnell" <sha...@knownormal.com>
Subject Re: Hbase master not starting
Date Mon, 04 Jan 2016 17:03:50 GMT
It's not there.  The directory listing was for the right directory, but the
namespace directory is not there.

Here it is one from directory level up:

[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
/hbase/data

drwxr-xr-x   - hdfs hadoop          0 2016-01-04 14:47 /hbase/data/hbase

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc

-rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta/.tmp

drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740

-rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/.regioninfo

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
/hbase/data/hbase/meta/1588230740/info

drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits

-rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid

========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!

On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> In your listing, meta table directory structure was shown.
>
> Please look under /hbase/data/hbase for namespace table.
>
> Cheers
>
> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <shaneo@knownormal.com>
> wrote:
>
> > I attempted to restore a backed-up "meta" directory from a copy made by
> > 'hbase hbck', so my directory structure may be messed up:
> >
> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
> -R
> > /hbase/data/hbase
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> > /hbase/data/hbase/meta
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tabledesc
> >
> > -rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tmp
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740
> >
> > -rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/.regioninfo
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
> > /hbase/data/hbase/meta/1588230740/info
> >
> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/recovered.edits
> >
> > -rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
> >
> > And I'm not seeing any "namespace" directory.
> >
> > Will check out the server now...
> >
> > Shane O.
> >
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Can you log onto the server hosting region
> > 82432aca9ede964943b40753cb64e808
> > > and see what happened ?
> > >
> > > See if the namespace table can be found under rootdir.
> > > e.g. assuming /apps/hbase/data is the rootdir, you should see something
> > > similar to the following:
> > >
> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
> > > Found 3 items
> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
> > > /apps/hbase/data/data/hbase/namespace/.tmp
> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:51
> > > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
> > >
> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <shaneo@knownormal.com
> >
> > > wrote:
> > >
> > > > Some progress...
> > > >
> > > > /hbase did NOT have either the hbase.id or hbase.version files so I
> > > > temporarily changed hbase.rootdir and started the master so the files
> > > would
> > > > be recreated elsewhere and copied them in.
> > > >
> > > > Now it starts fine, but my hbase tables are gone.  Specifically, I'm
> > > > getting this error:
> > > >
> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> > > >
> > > > Is my existing data toast, or is there a crafty way out of this?
> > > >
> > > > Thanks -
> > > >
> > > > Shane O.
> > > >
> > > > ========================
> > > > Shane O'Donnell
> > > > <
> > > >
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > > > >
> > > > 4819 Emperor Blvd., Ste 400
> > > > Durham, North Carolina 27703
> > > > tel: +1.424.262.KNOW x703
> > > > skype: shaneodonnell
> > > > email: shaneo@knownormal.com
> > > > ========================
> > > > :wq!
> > > >
> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
> > shaneo@knownormal.com>
> > > > wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > My cluster has been running perfectly until the other day when I
> > found
> > > it
> > > > > down.
> > > > >
> > > > > The error seems to be related not being able to get the ClusterID
> > from
> > > > > zookeeper, but I'm stumped as to what to do about it.  This seems
> to
> > be
> > > > the
> > > > > relevant part of the master's log:
> > > > >
> > > > >      http://pastebin.com/C3iaxM3p
> > > > >
> > > > > starting at line 235 (also highlighted).
> > > > >
> > > > > Any help is appreciated!
> > > > >
> > > > > Thanks -
> > > > >
> > > > > Shane O.
> > > > > ========================
> > > > > Shane O'Donnell
> > > > >
> > > > > <
> > > >
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > > > >
> > > > > 4819 Emperor Blvd., Ste 400
> > > > > Durham, North Carolina 27703
> > > > > tel: +1.424.262.KNOW x703
> > > > > skype: shaneodonnell
> > > > > email: shaneo@knownormal.com
> > > > > ========================
> > > > > :wq!
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message