hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Menges <dejan.men...@gmail.com>
Subject Re: Hbase master not starting
Date Tue, 05 Jan 2016 09:23:05 GMT
Hi Shane,

What was our case - at some point we couldn't bring up HBase anymore having
exactly same issues as you did - not being able to open hbase:namespace. On
HDFS I also couldn't find hbase:namespace anymore for that specific
database, so what I did was next:

- Let's say that your hbase.root was on your hdfs://somewhere:8020/hbase
- Back up all HBase related files on HDFS to different folder
- Change hbase.rootdir and zookeeper.znode.parent to something new, like
hbasetemp
- Start hbase, it will set up empty namespace and meta for you, with no
tables, on your new location
- Copy first table from ${old_hbase_root}/data/default/${table_name} to
${new_hbase_root}/data/default/
- Run hbck until it's saying all is good. You will most likely have to run
it multiple times with different options. Hopefully, if your tables are not
corrupted, and you have just issue with namespace, hbck will rebuild just
meta and namespace, and you will be good.
- Repeat this for every table, verify your data is still there in the end
of course. In the end if all is good you should have your tables recovered.
If you really want, you can take HBase down again, rename two parameters
from above and rename hbase.root on HDFS and restart HBase again, to have
stuff up and running on the old location.

Just to say it again: this was desperate man try, and in our case it
worked. Ok, we also wanted to test if it will work, but this wasn't
production at all, but wasn't also environment where we could loose any
data.

Good luck!

On Mon, Jan 4, 2016 at 9:42 PM Shane O'Donnell <shaneo@knownormal.com>
wrote:

> Thanks for this, Dejan!
>
> Where did you have your tables to move them in?  I know the names of all of
> my tables, but I'm not sure how I'd go about "copying my table to the new
> Hbase location".  Can you explain?
>
> Shane O.
>
> ========================
> Shane O'Donnell
> <https://www.knownormal.com/>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 9:44 AM, Dejan Menges <dejan.menges@gmail.com>
> wrote:
>
> > I had once exactly this situation, and was able (but that was some
> > development database, so wasn't super production big deal) to get over it
> > with:
> >
> > - Stop HBase
> > - Copy your table to your new HBase location
> > - Start your HBase
> > - Run HBCK until it says all is good and you are sure you have your data
> > - Repeat for every table you want to move.
> >
> > I'm quite sure this is not recommended way for anything, but that was the
> > only way for me once I hit hbase:namespace issue like yours...
> >
> > Good luck!
> >
> > On Mon, Jan 4, 2016 at 3:37 PM Shane O'Donnell <shaneo@knownormal.com>
> > wrote:
> >
> > > Some progress...
> > >
> > > /hbase did NOT have either the hbase.id or hbase.version files so I
> > > temporarily changed hbase.rootdir and started the master so the files
> > would
> > > be recreated elsewhere and copied them in.
> > >
> > > Now it starts fine, but my hbase tables are gone.  Specifically, I'm
> > > getting this error:
> > >
> > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> > >
> > > Is my existing data toast, or is there a crafty way out of this?
> > >
> > > Thanks -
> > >
> > > Shane O.
> > >
> > > ========================
> > > Shane O'Donnell
> > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
> shaneo@knownormal.com>
> > > wrote:
> > >
> > > > Hi -
> > > >
> > > > My cluster has been running perfectly until the other day when I
> found
> > it
> > > > down.
> > > >
> > > > The error seems to be related not being able to get the ClusterID
> from
> > > > zookeeper, but I'm stumped as to what to do about it.  This seems to
> be
> > > the
> > > > relevant part of the master's log:
> > > >
> > > >      http://pastebin.com/C3iaxM3p
> > > >
> > > > starting at line 235 (also highlighted).
> > > >
> > > > Any help is appreciated!
> > > >
> > > > Thanks -
> > > >
> > > > Shane O.
> > > > ========================
> > > > Shane O'Donnell
> > > >
> > > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > > >
> > > > 4819 Emperor Blvd., Ste 400
> > > > Durham, North Carolina 27703
> > > > tel: +1.424.262.KNOW x703
> > > > skype: shaneodonnell
> > > > email: shaneo@knownormal.com
> > > > ========================
> > > > :wq!
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message