hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cecile, Adam" <Adam.Cec...@hitec.lu>
Subject RE: Table is disabled an no way to get it back online
Date Wed, 23 Nov 2016 07:13:07 GMT
Hello,

Thanks a lot, the table is back online. One last question ? Can you provide a log pattern
to spot this ? Just in case it occurs again ;-)

Regards, Adam.
________________________________________
De : Matteo Bertozzi <theo.bertozzi@gmail.com>
Envoyé : mardi 22 novembre 2016 19:48
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

hadoop fs -rmr /hbase/MasterProcWALs

Matteo


On Tue, Nov 22, 2016 at 10:42 AM, Cecile, Adam <Adam.Cecile@hitec.lu> wrote:

> Can you explain me how to delete masterwals directory?
>
>
>
> Sent from my Samsung device
>
>
> -------- Original message --------
> From: Matteo Bertozzi <theo.bertozzi@gmail.com>
> Date: 22/11/2016 19:12 (GMT+01:00)
> To: user@hbase.apache.org
> Subject: Re: Table is disabled an no way to get it back online
>
> I don't think this has anything to do with HBASE-13415 or the bugfix it
> related to it i'm working on.
>
> this is probably the usual case of mismatch state with zk.  Disable is
> saying that the table is already not disabled.
> so, enableTable() is the one that should give any exception in case. but I
> don't see any.
> I suggest to just drop the MasterWALs directory, drop the znode
> /hbase/table/sentinel-meta, restart the master and try disable and then
> enable. that should bring you back with the table online
>
> Matteo
>
>
> On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> wrote:
>
> > Thanks for everything. As you said, this bug is supposed to be fixed in
> > 1.2.0. Matteo is reading this list as well ?
> >
> > Regards, Adam.
> > ________________________________________
> > De : Ted Yu <yuzhihong@gmail.com>
> > Envoyé : mardi 22 novembre 2016 19:00
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > Please take a look at HBASE-13415
> >
> > From the log, you're using hbase 1.2.0 already. But I heard there is a
> > subtle bug which is being fixed.
> >
> > Matteo is the person with best knowledge in this regard.
> >
> > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> > wrote:
> >
> > > Another one, because I'm not sure the log is overwritten when
> restarting.
> > > This one has been cleared before service start.
> > > ________________________________________
> > > De : Cecile, Adam <Adam.Cecile@hitec.lu>
> > > Envoyé : mardi 22 novembre 2016 18:42
> > > À : user@hbase.apache.org
> > > Objet : RE: Table is disabled an no way to get it back online
> > >
> > > Hello,
> > >
> > > Sadly I could not use the webui, it killed my firefox (probably way too
> > > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > > minutes running !!)
> > >
> > > Best regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yuzhihong@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 17:05
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > In log4j.properties :
> > >
> > > log4j.logger.org.apache.hadoop.hbase=DEBUG
> > >
> > > On master UI, you can select the Procedures tab. Pastebin what you see
> > > (text is enough).
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> > > wrote:
> > >
> > > > Hey Ted,
> > > >
> > > > Thank you. Heading home right now but I'll start the laptop again.
> Not
> > > > sure exactly how I should turn debug log on so if you have the
> > > information
> > > > it'd be appreciated, otherwise I'll look at the xml files.
> > > >
> > > > Regards, Adam.
> > > > ________________________________________
> > > > De : Ted Yu <yuzhihong@gmail.com>
> > > > Envoyé : mardi 22 novembre 2016 15:46
> > > > À : user@hbase.apache.org
> > > > Objet : Re: Table is disabled an no way to get it back online
> > > >
> > > > Master log contained entries in the following form:
> > > >
> > > > 2016-11-22 13:13:41,836 INFO  [ProcedureExecutor-3]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43538 owner=hbase         state=ROLLEDBACK
> > > > exec-time=242hrs, 10mins, 28.896sec
> > > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > > sentinel-meta
> > > >
> > > > Note the procedure Id was around 43000, far lower than 147464.
> > > >
> > > > Can you turn debug log on and repost master log ?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > >
> > > > > We're having a table stuck in disabled state. First I'd like to
> start
> > > > with
> > > > > what I tried already:
> > > > >
> > > > >
> > > > > * Restart all machines involved in HBase cluster
> > > > >
> > > > > * hbase hbck with varios arguments
> > > > >
> > > > > * hdfs fsck
> > > > >
> > > > > * Purge ZK /hbase and restart masters
> > > > >
> > > > >
> > > > > Now more details anout what happens:
> > > > >
> > > > > * When enabling from hbase shell:
> > > > >
> > > > >
> > > > > hbase(main):002:0> enable "sentinel-meta"
> > > > > ERROR: The procedure 147464 is still running
> > > > >
> > > > >
> > > > > The task ID changes every time I run the command so I think it's
> > > talking
> > > > > about itself (and it gets stuck for a while before saying anything)
> > > > >
> > > > >
> > > > > In the log, all I can see is:
> > > > >
> > > > > 2016-11-22 13:10:50,776 INFO  [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:50,781 INFO  [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > > 2016-11-22 13:10:51,084 INFO  [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:51,088 INFO  [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > >
> > > > >
> > > > > Please also find attached a complete log from startup to shutdown
> on
> > a
> > > > > single active master. You'll see the table is found as well as the
> > > > regions
> > > > > but it gets deactivated with no reason.
> > > > >
> > > > >
> > > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Adam.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
View raw message