Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
DomainKey-Signature: a=rsa-sha1; c=simple; d=hitec.lu; h=from:to:subject
	:date:message-id:references:in-reply-to:content-type
	:content-transfer-encoding:mime-version; q=dns; s=postfix; b=O+I
	RZkRXXkowFcBVnoWkZTIe5SzjQ63SnP1SPvl5qT6DL6j9nj6Wvq+f9gcLFxRK02t
	W7BoW8/o5phnlfypu6b/e34r1uJG7Vc9UifXM3FqSU+7iBEw/ZWrV1g/QwWoRDfT
	dVm05AC2Nv1npk/GYoiTN1BHTnxvnu7oT41Q2XSY=
From: "Cecile, Adam" <Adam.Cecile@hitec.lu>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: RE: Table is disabled an no way to get it back online
Thread-Topic: Table is disabled an no way to get it back online
Thread-Index: AQHSRLkFGIq5E+OIQkaC6CMfk1SM0aDlBCSAgAAY3XQ=
Date: Tue, 22 Nov 2016 15:16:39 +0000
Message-ID: <a9acb651368c4022ada245cf64f8c89b@HI-SRV-05.hitec.lan>
References: <c007927211df4d92be61931fe20451e4@HI-SRV-05.hitec.lan>,<CALte62wQTOV8nRmzC4Aknq0jSXcJGgokHJSfcXSTyBfM++fc5g@mail.gmail.com>
In-Reply-To: <CALte62wQTOV8nRmzC4Aknq0jSXcJGgokHJSfcXSTyBfM++fc5g@mail.gmail.com>
Accept-Language: fr-FR, lb-LU, en-US
Content-Language: fr-FR
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
archived-at: Tue, 22 Nov 2016 15:17:24 -0000

Hey Ted,=0A=
=0A=
Thank you. Heading home right now but I'll start the laptop again. Not sure=
 exactly how I should turn debug log on so if you have the information it'd=
 be appreciated, otherwise I'll look at the xml files.=0A=
=0A=
Regards, Adam.=0A=
________________________________________=0A=
De : Ted Yu <yuzhihong@gmail.com>=0A=
Envoy=E9 : mardi 22 novembre 2016 15:46=0A=
=C0 : user@hbase.apache.org=0A=
Objet : Re: Table is disabled an no way to get it back online=0A=
=0A=
Master log contained entries in the following form:=0A=
=0A=
2016-11-22 13:13:41,836 INFO  [ProcedureExecutor-3]=0A=
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure=0A=
(table=3Dsentinel-meta) id=3D43538 owner=3Dhbase         state=3DROLLEDBACK=
=0A=
exec-time=3D242hrs, 10mins, 28.896sec=0A=
exception=3Dorg.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta=
=0A=
=0A=
Note the procedure Id was around 43000, far lower than 147464.=0A=
=0A=
Can you turn debug log on and repost master log ?=0A=
=0A=
Thanks=0A=
=0A=
On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu> wrote:=
=0A=
=0A=
> Hello,=0A=
>=0A=
>=0A=
> We're having a table stuck in disabled state. First I'd like to start wit=
h=0A=
> what I tried already:=0A=
>=0A=
>=0A=
> * Restart all machines involved in HBase cluster=0A=
>=0A=
> * hbase hbck with varios arguments=0A=
>=0A=
> * hdfs fsck=0A=
>=0A=
> * Purge ZK /hbase and restart masters=0A=
>=0A=
>=0A=
> Now more details anout what happens:=0A=
>=0A=
> * When enabling from hbase shell:=0A=
>=0A=
>=0A=
> hbase(main):002:0> enable "sentinel-meta"=0A=
> ERROR: The procedure 147464 is still running=0A=
>=0A=
>=0A=
> The task ID changes every time I run the command so I think it's talking=
=0A=
> about itself (and it gets stuck for a while before saying anything)=0A=
>=0A=
>=0A=
> In the log, all I can see is:=0A=
>=0A=
> 2016-11-22 13:10:50,776 INFO  [ProcedureExecutor-0]=0A=
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure=
=0A=
> (table=3Dsentinel-meta) id=3D43220 owner=3Dhbase state=3DROLLEDBACK=0A=
> exec-time=3D242hrs, 52mins, 7.454sec exception=3Dorg.apache.hadoop.hbase.=
TableNotEnabledException:=0A=
> sentinel-meta=0A=
> 2016-11-22 13:10:50,781 INFO  [ProcedureExecutor-0] procedure.DisableTabl=
eProcedure:=0A=
> Table sentinel-meta isn't enabled; skipping disable=0A=
> 2016-11-22 13:10:51,084 INFO  [ProcedureExecutor-0]=0A=
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure=
=0A=
> (table=3Dsentinel-meta) id=3D43221 owner=3Dhbase state=3DROLLEDBACK=0A=
> exec-time=3D242hrs, 51mins, 42.288sec exception=3Dorg.apache.hadoop.hbase=
.TableNotEnabledException:=0A=
> sentinel-meta=0A=
> 2016-11-22 13:10:51,088 INFO  [ProcedureExecutor-0] procedure.DisableTabl=
eProcedure:=0A=
> Table sentinel-meta isn't enabled; skipping disable=0A=
>=0A=
>=0A=
> Please also find attached a complete log from startup to shutdown on a=0A=
> single active master. You'll see the table is found as well as the region=
s=0A=
> but it gets deactivated with no reason.=0A=
>=0A=
>=0A=
> Thanks a lot for your help, we're kinda running out if ideas here.=0A=
>=0A=
>=0A=
> Best regards,=0A=
>=0A=
>=0A=
> Adam.=0A=
>=0A=
>=0A=
>=0A=