hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Gupta Tankala <tvi...@readypulse.com>
Subject org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 3 actions: WrongRegionException:
Date Tue, 27 Sep 2011 06:26:55 GMT
Hi,
I have seen similar issue being handled here but i didn't fully understand
how to fix it in my case. this is the earlier thread -
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/18410

i am seeing the below issues in my client -
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
3 ac
tions: WrongRegionException: 3 times, servers with issues:
ip-10-32-61-60.ec2.in
ternal:44911,
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatch(HConnectionManager.java:1220)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatchOfPuts(HConnectionManager.java:1234)
        at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
...

also, hbase hbck -details gives the following errors. -fix doesn't do
anything. neither does check_meta.rb.

ERROR: Region
AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
not deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3af08f3ba88d7c2785fd089702f89241
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/a522e78c7d018547b8979ed6fd381358
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/b89f7bb47e53ea9cc729c86978a7327c
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/fd7ce07003bfd883f53a210bb0985065
on HDFS, but not listed in META or deployed on any region server.
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
ERROR: Found inconsistency in table AkStats
Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  ip-10-32-61-60.ec2.internal:55915
  .META. is okay.
    Number of regions: 1
    Deployed on:  ip-10-32-61-60.ec2.internal:55915
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
Table AkStats is inconsistent.
    Number of regions: 133
    Deployed on:  ip-10-32-61-60.ec2.internal:55915
7 inconsistencies detected.
Status: INCONSISTENT

my production system is down on its knees and I need to get unblocked asap.
any help will be highly appreciated.

thanks
vinod

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message