hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Napoli <jnap...@swipely.com>
Subject Re: HBase hung, lost metadata for tables
Date Tue, 11 Oct 2011 16:26:48 GMT
We're running in distributed mode. The rootdir is
hdfs://s-hbase-master/hbase. The HDFS files appear to be intact (we'll try
to recover the tables tomorrow). We are running java "1.6.0_17-b04". We're
attempting to reproduce the problem and will do a thread dump if successful.

Thanks,
Joshua

On Fri, Oct 7, 2011 at 11:53 PM, Stack <stack@duboce.net> wrote:

> Is this standalone? Are you using default /tmp location for your data?
>
> Please thread dump  your hung server the next time.  Save a jstack of
> the process or send a kill -QUIT PID and send us the .out file.
>
> What JVM?
>
> Thanks,
> St.Ack
>
> On Fri, Oct 7, 2011 at 11:58 AM, Joshua Napoli <jnapoli@swipely.com>
> wrote:
> > I'm using HBase 0.90.4 and encountered a problem where:
> >
> >   1. HBase became unable to processes requests. REST client requests
> >   time-out and HBase commands from HBase shell would hang.
> >   2. After restarting the cluster, almost all of my tables were missing
> >   (but the HDFS files remain).
> >
> >
> > While the cluster was hung, the REST logs were repeating (for the same
> > tables over and over):
> > "2011-10-05 10:09:54,801 DEBUG org.apache.hadoop.hbase.client.HBaseAdmin:
> > Sleeping= 32000ms, waiting for all regions to be disabled in ...".
> >
> > The messages list tables that were meant for deletion. However, many
> other
> > tables (not listed for disable/delete in the logs) were lost.
> >
> > What could cause the loss of table information? What is the proper
> sequence
> > for restarting a cluster that is no longer able to process HBase
> requests?
> >
> >
> > Thanks,
> > Josh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message