hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mesika, Asaf" <asaf.mes...@gmail.com>
Subject Re: Region is not online: .META.,,1
Date Thu, 08 Nov 2012 14:31:37 GMT
More updates in case it can help:

In the UI I get the following regions mapping for .META.:
Table Regions

Name	Region Server	Start Key	End Key	Requests
.META.,,1.1028785192	172.25.98.66:11401	-		




When I run 'hbase hbck' I fail, with the following error which indicates is thinks this region
is placed in a different server (65 and not 66)

12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192
for tableName=.META. from cache because of 
12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location
for .META.,,1.1028785192 is 172.25.98.65:11411
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=10, exceptions:
Thu Nov 08 14:22:23 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@39ed1b0b, org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1


How can there be such a mismatch?




On Nov 7, 2012, at 8:08 AM, Mesika, Asaf wrote:

> After restart, running hbck results in:
> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647
rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
> 
> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
> Exception in thread "main" java.lang.NullPointerException
> 	at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
> 	at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
> 	at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> 	at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
> 
> HBase is version 0.94.0
> 
> 
> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
> 
>> Hi,
>> 
>> I have a cluster with 10 RS and 1 Master.
>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>> 
>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2,
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException:
Region is not online: .META.,,1
>> 
>> 	at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>> 	at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>> 	at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>> 	at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>> 	at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>> 	at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>> 	at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> 	at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>> 
>> 
>> Anyone has any idea how to fix it without the obvious restart?
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message