Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Date: Sun, 15 Nov 2015 08:29:56 +0000 (UTC)
From: Sumit Nigam <sumit_only@yahoo.com.INVALID>
Reply-To: Sumit Nigam <sumit_only@yahoo.com>
To: Ted Yu <yuzhihong@gmail.com>,
	"user@hbase.apache.org" <user@hbase.apache.org>
Message-ID: <1857428539.5122369.1447576196480.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: 
 <CALte62w=qicpLhK11kX-1EP0fT=E2DvpEwU80UFH3gR7YCS8DQ@mail.gmail.com>
References: 
 <CALte62w=qicpLhK11kX-1EP0fT=E2DvpEwU80UFH3gR7YCS8DQ@mail.gmail.com>
Subject: Re: About exceptions
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_5122368_375003580.1447576196466"

------=_Part_5122368_375003580.1447576196466
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Ted,
Thanks for your reply. I am using Hbase 0.98.14. I have used hbck, but for =
some (unknown) reason it has not always resolved inconsistencies.=C2=A0
I have been able to get around these issues so far by deleting ZK entries f=
or the offending table and restarting Hbase. But I am not sure what causes =
them in the first place and if I can avoid those issues through code or not=
. Also, upon getting these exceptions is it a good idea to retry the operat=
ion. I think Hbase fails only after it exhausts its attempts so retrying ma=
y not be helpful?

Here are 3 logs snippets:
1.=C2=A0TableNamespaceManager isn't ready to serve:
Fri Nov 13 17:47:19 IST 2015, org.apache.hadoop.hbase.client.RpcRetryingCal=
ler@44726f67,org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.=
IOException):java.io.IOException: org.apache.hadoop.hbase.master.TableNames=
paceManager isn'tready to serve

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 atorg.apache.hadoop.hbase.master=
.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:112)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 atorg.apache.hadoop.hbase.master=
.TableNamespaceManager.list(TableNamespaceManager.java:211)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 atorg.apache.hadoop.hbase.master=
.HMaster.listNamespaceDescriptors(HMaster.java:3473)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 atorg.apache.hadoop.hbase.master=
.HMaster.listNamespaceDescriptors(HMaster.java:3367)


2. TableExistsException:
Caused by: org.apache.hadoop.hbase.TableExistsException: org.apache.hadoop.=
hbase.TableExistsException: ldmns:exDocStoreat org.apache.hadoop.hbase.mast=
er.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:3=
00)at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.execute=
FromState(CreateTableProcedure.java:106)at org.apache.hadoop.hbase.master.p=
rocedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58=
)...
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetr=
yingCaller.java:90)at org.apache.hadoop.hbase.client.HBaseAdmin.executeCall=
able(HBaseAdmin.java:3403)at org.apache.hadoop.hbase.client.HBaseAdmin.crea=
teTableAsync(HBaseAdmin.java:632)at org.apache.hadoop.hbase.client.HBaseAdm=
in.createTable(HBaseAdmin.java:523)

3. TableNotEnabledException:
Caused by: org.apache.hadoop.hbase.TableNotEnabledException: ldmns:DataDoma=
in_stage is disabled. at org.apache.hadoop.hbase.client.HConnectionManager$=
HConnectionImplementation.relocateRegion(HConnectionManager.java:1139) at o=
rg.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.=
getRegionLocation(HConnectionManager.java:963) at org.apache.hadoop.hbase.c=
lient.RegionServerCallable.prepare(RegionServerCallable.java:74) at org.apa=
che.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller=
.java:114) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833) at=
 org.apache.hadoop.hbase.client.HTable.get(HTable.java:810) at org.apache.h=
adoop.hbase.client.HTable.get(HTable.java:842) at com.thinkaurelius.titan.d=
iskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStor=
e.java:155)
      From: Ted Yu <yuzhihong@gmail.com>
 To: "user@hbase.apache.org" <user@hbase.apache.org>; Sumit Nigam <sumit_on=
ly@yahoo.com>=20
 Sent: Sunday, November 15, 2015 10:50 AM
 Subject: Re: About exceptions
  =20
bq.=C2=A0TableNotEnabledExceptionTableNotFoundExceptionIOException
Can you show log snippets where these exceptions occurred ?Which release of=
 hbase are you using ?
Have you run hbck to repair the inconsistencies ?
See=C2=A0http://hbase.apache.org/book.html#hbck.in.depth
Cheers


On Sat, Nov 14, 2015 at 8:42 PM, Sumit Nigam <sumit_only@yahoo.com.invalid>=
 wrote:

Hi,
There are some exceptions which I face intermittently with Hbase and I thou=
ght some help from experts online can really help me. These are:
TableNotEnabledExceptionTableNotFoundExceptionIOException - TableNamespaceM=
anager isn't ready to serve

One of the reasons I can see for this seems to be zookeeper and Hbase/ Hdfs=
 data being out of sync due to an unclean shutdown.=C2=A0
So, my questions are these:
1. Are these exceptions only related to unclean shutdowns?2. Do I need to e=
xplicitly handle them and retry the operation again because they also seem =
to indicate that it is some race condition between trying to access a table=
 vs Hbase enabling them?
Any help is greatly appreciated.
Thanks,Sumit


------=_Part_5122368_375003580.1447576196466--