hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: .META. stuck when the network switch down
Date Thu, 15 Mar 2012 20:50:25 GMT
First, upgrade to a hadoop version that supports append.

When you disable/drop the table, do you still see CLOUD_201203 entries
in .META. (using the shell)? It seem there's something odd in there.

Also look for odd exceptions in the master log when you create the
table, and make sure it assigns the newly created region.

J-D

On Thu, Mar 15, 2012 at 8:21 AM, Weihua JIANG <weihua.jiang@gmail.com> wrote:
> We are using hadoop 0.20.2 + hbase 0.90.3.
> The network switch of our cluster was down for some time. After restore,
> several data nodes and RSes (about half) were down. After restart them, the
> cluster was online.
> However, our writers to hbase always reported following info:
> 2012-03-15 22:50:17,313 WARN
> [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
> - Encountered problems when prefetch META table:
> java.io.IOException: HRegionInfo was null or empty in Meta for
> CLOUD_201203,
> row=CLOUD_201203,14522035490150643060323210-9883,99999999999999
>        at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
>        at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:648)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:702)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1117)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1237)
>        at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
>        at com.huawei.cloud.output.HBaseWriter.put(HBaseWriter.java:59)
>        at com.huawei.cloud.output.HBaseWriter.run(HBaseWriter.java:102)
>        at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
>
> hbck reports lots of missing of region infos for CLOUD_201203 in .META.
> table.
>
> So, I disabled and dropped table CLOUD_201203 in hbase shell. Then,
> re-create the table using hbase shell.
>
> Initially, it looked normal with only one exception: the web page for this
> table showed NO region. But, hbck reported OK. So I started the writer
> programs to write data to this table. After importing about 500MB data,
> client again got above errors.
>
> So, it seems the table re-creation still did NOT fix the problem.
>
> What shall I do then?
>
> Thanks
> Weihua

Mime
View raw message