hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeba earnest <jebaearn...@yahoo.com>
Subject Inconsistency in Hbase table[Region not deployed on any region server]
Date Fri, 28 Feb 2014 15:14:55 GMT
In a hbase cluster, all the slave nodes got restarted. When I started Hbase services,
one of the table(test) become inconsistent.

HDFS some blocks were missing(hbase blocks). so it was in safe mode. I gave "safemode -leave"
Then hbase table (test)became inconsistent.

Below actions I performed


1. executed "hbase hbck" several times  
2 inconsistencies found for table "test"

2. hbase hbck -fixMeta -fixAssignments   
HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.',
STARTKEY = '1m\x00\x03\x1B\x15', ENDKEY = '', ENCODED = 4c213a47bba83c47075f21fec7c6d862,}

3. hbase hbck -repair    
HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.',
STARTKEY = '1m\x00\x03\x1B\x15', ENDKEY = '', ENCODED = 4c213a47bba83c47075f21fec7c6d862,}

4. I checked datanode logs parallely  
Log: org.apache.hadoop.hdfs.server.datanode.DataNode: opReadBlock BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846
received exception java.io.EOFException   
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.1.12, storageID=DS-831971799-192.168.1.12-50010-1391193910800,
infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-7f99a9de-258c-493c-9db0-46b9e84b4c12;nsid=1286773982;c=0):Got
exception while serving BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846
to /192.168.1.12:36127


5. Checked Namenode logs   
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ubuntu
(auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C
2014-02-28 14:13:15,738 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9000, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 10.10.242.31:42149:
error: java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C
java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1301) 

But, I am able to browse and download the file from HDFS. How can recover the data?
How can I make the "test" table consistent.


Regards,

Jeba
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message