hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: About exceptions
Date Sun, 15 Nov 2015 10:04:32 GMT
Sumit:
For #1, I have seen a similar issue (HBASE-14190, though on hbase 1.x
release).
If you have debug logging enabled, please pastebin relevant master log
snippet so that we can take a closer look.

For #2, I am bit confused - I didn't find CreateTableProcedure.java in 0.98
branch. To my knowledge, CreateTableProcedure is only in hbase 1 release.
Did you see the stack trace in master log ?

For #3, there could be various reasons a table was not enabled.
You can trace the table assignment in master log, check log from hbase:meta
server to see if you can find some clue.

bq. Hbase fails only after it exhausts its attempts so retrying may not be
helpful?

Your understanding should be correct.

I want to bring your attention to HBASE-12070 which helps you fix ZK
inconsistencies.

Cheers

On Sun, Nov 15, 2015 at 12:29 AM, Sumit Nigam <sumit_only@yahoo.com> wrote:

> Hi Ted,
>
> Thanks for your reply. I am using Hbase 0.98.14. I have used hbck, but for
> some (unknown) reason it has not always resolved inconsistencies.
>
> I have been able to get around these issues so far by deleting ZK entries
> for the offending table and restarting Hbase. But I am not sure what causes
> them in the first place and if I can avoid those issues through code or
> not. Also, upon getting these exceptions is it a good idea to retry the
> operation. I think Hbase fails only after it exhausts its attempts so
> retrying may not be helpful?
>
>
> Here are 3 logs snippets:
>
> 1. TableNamespaceManager isn't ready to serve:
>
> Fri Nov 13 17:47:19 IST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@44726f67,
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.*IOException*: org.apache.hadoop.hbase.master.*TableNamespaceManager
> isn't ready to serve*
>         at
> org.apache.hadoop.hbase.master.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:112)
>         at
> org.apache.hadoop.hbase.master.TableNamespaceManager.list(TableNamespaceManager.java:211)
>         at
> org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3473)
>         at
> org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3367)
>
>
>
> 2. TableExistsException:
>
> Caused by: org.apache.hadoop.hbase.TableExistsException:
> org.apache.hadoop.hbase.*TableExistsException: ldmns:exDocStore*
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:300)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:106)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)
> ...
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3403)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:632)
> at org.apache.hadoop.hbase.client.HBaseAdmin.*createTable*
> (HBaseAdmin.java:523)
>
>
> 3. TableNotEnabledException:
>
> Caused by: org.apache.hadoop.hbase.*TableNotEnabledException*:
> ldmns:DataDomain_stage is disabled.
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1139)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:963)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:74)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:810)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:842)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:155)
>
> ------------------------------
> *From:* Ted Yu <yuzhihong@gmail.com>
> *To:* "user@hbase.apache.org" <user@hbase.apache.org>; Sumit Nigam <
> sumit_only@yahoo.com>
> *Sent:* Sunday, November 15, 2015 10:50 AM
> *Subject:* Re: About exceptions
>
> bq. TableNotEnabledExceptionTableNotFoundExceptionIOException
>
> Can you show log snippets where these exceptions occurred ?
> Which release of hbase are you using ?
>
> Have you run hbck to repair the inconsistencies ?
>
> See http://hbase.apache.org/book.html#hbck.in.depth
>
> Cheers
>
>
>
> On Sat, Nov 14, 2015 at 8:42 PM, Sumit Nigam <sumit_only@yahoo.com.invalid
> > wrote:
>
> Hi,
> There are some exceptions which I face intermittently with Hbase and I
> thought some help from experts online can really help me. These are:
> TableNotEnabledExceptionTableNotFoundExceptionIOException -
> TableNamespaceManager isn't ready to serve
>
> One of the reasons I can see for this seems to be zookeeper and Hbase/
> Hdfs data being out of sync due to an unclean shutdown.
> So, my questions are these:
> 1. Are these exceptions only related to unclean shutdowns?2. Do I need to
> explicitly handle them and retry the operation again because they also seem
> to indicate that it is some race condition between trying to access a table
> vs Hbase enabling them?
> Any help is greatly appreciated.
> Thanks,Sumit
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message