hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Shcheklein <shchekl...@gmail.com>
Subject Name node log roller destroyed metadata?
Date Sun, 17 Apr 2016 05:30:21 GMT
Hi all,

HBase version: Version 0.94.18, re8f20e1020decc8f106e952f393f0484058068e2,
Wed Nov  4 20:57:33 GMT 2015

Amazon EMR

We got HBase namenode down recently with the "Namenode is not formatted
error" and had to restore everything from backups. Below is what we have in
the namenode log file a couple hours before that. Any ideas how can we
prevent this from happening in the future? Any ideas what did cause this
problem in the first place? Some error in hadoop hdfs? Any ideas how to
investigate it further. We would appreciate any help.

Full log link:
https://www.dropbox.com/s/ynmd9nq8vp3tt98/hadoop-hadoop-namenode-ip-172-31-25-236.log.2016-04-16-23.gz?dl=0

================================================================

2016-04-16 23:01:37,187 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Rolling edit logs
2016-04-16 23:01:37,187 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Ending log segmen
t 1
2016-04-16 23:01:37,188 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Number of transac
tions: 2004180 Total time for transactions(ms): 24698 Number of
transactions batched in Syncs: 8993 Number of syncs: 1540948 SyncTimes(ms):
4583119
2016-04-16 23:01:37,190 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Finalizi
ng edits file /hadoop/dfs-name/current/edits_inprogress_0000000000000000001
-> /hadoop/dfs-name/current/edits_0000000000000000001-0000000000002004180
2016-04-16 23:01:37,191 ERROR org.apache.hadoop.hdfs.server.common.Storage
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Error reported on storage directory Storage Directory /hadoop/dfs-name
2016-04-16 23:01:37,191 WARN org.apache.hadoop.hdfs.server.common.Storage
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
About to remove corresponding storage: /hadoop/dfs-name
2016-04-16 23:01:37,191 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Error: finalize log segment 1, 2004180 failed for (journal
JournalAndStream(mgr=FileJournalManager(root=/hadoop/dfs-name),
stream=null))
java.lang.IllegalStateException: Unable to finalize edits file
/hadoop/dfs-name/current/edits_inprogress_0000000000000000001
        at
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:137)
        at
org.apache.hadoop.hdfs.server.namenode.JournalSet$3.apply(JournalSet.java:223)
        at
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:359)
        at
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:218)
        at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1168)
        at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1097)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1190)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5376)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4356)
        at java.lang.Thread.run(Thread.java:745)
2016-04-16 23:01:37,191 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Disabling journal
JournalAndStream(mgr=FileJournalManager(root=/hadoop/dfs-name), stream=null)
2016-04-16 23:01:37,191 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Error: finalize log segment 1, 2004180 failed for too many journals
2016-04-16 23:01:37,191 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Starting log segment at 2004181
2016-04-16 23:01:37,192 WARN
org.apache.hadoop.hdfs.server.namenode.FileJournalManager
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553):
Unable to start log segment 2004181 at
/hadoop/dfs-name/current/edits_inprogress_0000000000002004181:
/hadoop/dfs-name/current/edits_inprogress_0000000000002004181 (No such file
or directory)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message