hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Ranjan <vikas.ran...@outlook.com>
Subject NN recovery from metadata backup
Date Fri, 06 Jun 2014 02:32:23 GMT
Hi,I am trying NN metadata recovery.  I have taken backup of Namenode and Journal node meta
data . It contains edit logs and fsimages.
There are two NNs in my system. I take backup of metadata on both NNs (hdfs metadata &
QJM metadata) at regular frequency. I want to test recovery procedure in a worst case scenario.
Assume both the NNs and Journal node are down with the metadata completely deleted.
I want to recover NN metadata from backup and start NN. I know that there could be a data
loss as the latest changes done after backup would be missing.
Do you think such a scenario is possible/feasible ?I am facing some issues related to txn
id mismatch, committed txn id. Please tell if there is a solution for the same.
Steps tried:Take metadata backup of NN and QJM.Do some hdfs file operations (create new files).Stop
NN and Journal node on both the machines.Delete metadata from /data/hdfs and journal directories.Restore
Fsimages from backup (taken some time back). Start NN. It fails with below exception.Alternative:
Restore all the edit logs and fsimage to both hdfs and qjm directories and start NN but still
it fails.
Both the NNs are down and I can't bring up. I don't want to format hdfs as it will change
Cluster ID and the backup won't be usable.

Exceptions:There appears to be a gap in the edit log.  We expected txid 71453, but got txid
71466Client trying to move committed txid backward from 71599 to 71453recoverUnfinalizedSegments
failed for required journal. Decided to synchronize log to startTxId: 71453 but logger
had seen txid 71599 committed 		 	   		  
View raw message