hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cecile, Adam" <Adam.Cec...@hitec.lu>
Subject RE: Table is disabled an no way to get it back online
Date Tue, 22 Nov 2016 18:42:30 GMT
Can you explain me how to delete masterwals directory?



Sent from my Samsung device


-------- Original message --------
From: Matteo Bertozzi <theo.bertozzi@gmail.com>
Date: 22/11/2016 19:12 (GMT+01:00)
To: user@hbase.apache.org
Subject: Re: Table is disabled an no way to get it back online

I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.

this is probably the usual case of mismatch state with zk.  Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online

Matteo


On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Adam.Cecile@hitec.lu> wrote:

> Thanks for everything. As you said, this bug is supposed to be fixed in
> 1.2.0. Matteo is reading this list as well ?
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yuzhihong@gmail.com>
> Envoyé : mardi 22 novembre 2016 19:00
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Please take a look at HBASE-13415
>
> From the log, you're using hbase 1.2.0 already. But I heard there is a
> subtle bug which is being fixed.
>
> Matteo is the person with best knowledge in this regard.
>
> On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> wrote:
>
> > Another one, because I'm not sure the log is overwritten when restarting.
> > This one has been cleared before service start.
> > ________________________________________
> > De : Cecile, Adam <Adam.Cecile@hitec.lu>
> > Envoyé : mardi 22 novembre 2016 18:42
> > À : user@hbase.apache.org
> > Objet : RE: Table is disabled an no way to get it back online
> >
> > Hello,
> >
> > Sadly I could not use the webui, it killed my firefox (probably way too
> > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > minutes running !!)
> >
> > Best regards, Adam.
> > ________________________________________
> > De : Ted Yu <yuzhihong@gmail.com>
> > Envoyé : mardi 22 novembre 2016 17:05
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > In log4j.properties :
> >
> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> >
> > On master UI, you can select the Procedures tab. Pastebin what you see
> > (text is enough).
> >
> > Thanks
> >
> > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> > wrote:
> >
> > > Hey Ted,
> > >
> > > Thank you. Heading home right now but I'll start the laptop again. Not
> > > sure exactly how I should turn debug log on so if you have the
> > information
> > > it'd be appreciated, otherwise I'll look at the xml files.
> > >
> > > Regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yuzhihong@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 15:46
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > Master log contained entries in the following form:
> > >
> > > 2016-11-22 13:13:41,836 INFO  [ProcedureExecutor-3]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43538 owner=hbase         state=ROLLEDBACK
> > > exec-time=242hrs, 10mins, 28.896sec
> > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > sentinel-meta
> > >
> > > Note the procedure Id was around 43000, far lower than 147464.
> > >
> > > Can you turn debug log on and repost master log ?
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > > We're having a table stuck in disabled state. First I'd like to start
> > > with
> > > > what I tried already:
> > > >
> > > >
> > > > * Restart all machines involved in HBase cluster
> > > >
> > > > * hbase hbck with varios arguments
> > > >
> > > > * hdfs fsck
> > > >
> > > > * Purge ZK /hbase and restart masters
> > > >
> > > >
> > > > Now more details anout what happens:
> > > >
> > > > * When enabling from hbase shell:
> > > >
> > > >
> > > > hbase(main):002:0> enable "sentinel-meta"
> > > > ERROR: The procedure 147464 is still running
> > > >
> > > >
> > > > The task ID changes every time I run the command so I think it's
> > talking
> > > > about itself (and it gets stuck for a while before saying anything)
> > > >
> > > >
> > > > In the log, all I can see is:
> > > >
> > > > 2016-11-22 13:10:50,776 INFO  [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:50,781 INFO  [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > > 2016-11-22 13:10:51,084 INFO  [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:51,088 INFO  [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > >
> > > >
> > > > Please also find attached a complete log from startup to shutdown on
> a
> > > > single active master. You'll see the table is found as well as the
> > > regions
> > > > but it gets deactivated with no reason.
> > > >
> > > >
> > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Adam.
> > > >
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message