hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "patrick white (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3771) Namenode can't restart due to corrupt edit logs, timing issue with shutdown and edit log rolling
Date Mon, 13 Aug 2012 01:52:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432883#comment-13432883

patrick white commented on HDFS-3771:

Hi Todd, do you mind if i ask your opinion on the potential impact of this issue, and possible
mitigation for it? So far we've not been able to reproduce the issue, either in the nightly
regression testing or in specific tests. That, combined with regression history, would tend
to indicate this is very unlikely to occur operationally, however if it would occur the affects
could be severe. The NN restart should not be difficult once the cause is identified as corrupt
edits, it should come up once the corrupt logs are cleared. Is that true or am i missing an
event sequence tracking that would need to be corrected? 

The larger concern should be for data loss. Based on what happened in this case it appears
that any pending txids would be lost, unless the edit logs could be manually repaired. The
filesystem would be intact, only minus the changes from the outstanding edit events, does
that sound correct?
> Namenode can't restart due to corrupt edit logs, timing issue with shutdown and edit
log rolling
> ------------------------------------------------------------------------------------------------
>                 Key: HDFS-3771
>                 URL: https://issues.apache.org/jira/browse/HDFS-3771
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.23.3, 2.0.0-alpha
>         Environment: QE, 20 node Federated cluster with 3 NNs and 15 DNs, using Kerberos
based security
>            Reporter: patrick white
>            Priority: Critical
> Our 0.23.3 nightly HDFS regression suite encountered a particularly nasty issue recently,
which resulted in the cluster's default Namenode being unable to restart, this was on a 20
node Federated cluster with security. The cause appears to be that the NN was just starting
to roll its edit log when a shutdown occurred, the shutdown was intentional to restart the
cluster as part of an automated test.
> The tests that were running do not appear to be the issue in themselves, the cluster
was just wrapping up an adminReport subset and this failure case has not reproduce so far,
nor was it failing previously. It looks like a chance occurrence of sending the shutdown just
as the edit log roll was begun.
> From the NN log, the following sequence is noted:
> 1. an InvalidateBlocks operation had completed
> 2. FSNamesystem: Roll Edit Log from [Secondary Namenode IPaddr]
> 3. FSEditLog: Ending log segment 23963
> 4. FSEditLog: Starting log segment at 23967
> 4. NameNode: SHUTDOWN_MSG
> => the NN shuts down and then is restarted...
> 5. FSImageTransactionalStorageInspector: Logs beginning at txid 23967 were are all in-progress
> 6. FSImageTransactionalStorageInspector: Marking log at /grid/[PATH]/edits_inprogress_0000000000000023967
as corrupt since it has no transactions in it.
> 7. NameNode: Exception in namenode join [main]java.lang.IllegalStateException: No non-corrupt
logs for txid 23967
> => NN start attempts continue to cycle trying to restart but can't, failing on the
same exception due to lack of non-corrupt edit logs
> If observations are correct and issue is from shutdown happening as edit logs are rolling,
does the NN have an equivalent to the conventional fs 'sync' blocking action that should be
called, or perhaps has a timing hole?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message