hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1
Date Tue, 28 Aug 2012 07:09:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443002#comment-13443002
] 

Tsz Wo (Nicholas), SZE commented on HDFS-3540:
----------------------------------------------

If the edit log is not corrupted, both recovery mode and edit log toleration are not useful.
 Note that recovery mode here means recovery mode in branch-1 but not the one in trunk.

When an edit log is corrupted, NN cannot start up normally.  We compare recovery mode and
edit log toleration below.

*Recovery Mode*
- "Recovery" here means starting NN with a corrupted edit log.  It is unable to recover the
corrupted edit log or transactions.
- There is a namenode command option "hadoop namenode -recover" to enter recovery mode.
- When reading the first corrupted transaction in the edit log, it prompts the admin to either
"stop reading" or "quit without saving".
- If "stop reading" is selected, NN ignores the remaining edit log (from the first corrupted
transaction to the end of the edit log) and then starts up as usual.
- There is a "-force" option to FORCE_FIRST_CHOICE, i.e. it is a non-interactive mode.
- If there is a stray OP_INVALID byte, it could be misinterpreted as an end-of-log and lead
to silent data loss.  Recovery Mode does not help.
(Please help out if I have missed anything.)

*Edit Log Toleration*
- It has a conf property "dfs.namenode.edits.toleration.length" for setting the toleration
length.
- The default toleration length is -1, i.e. disable it.  The feature is enabled when the value
>= 0.
- When the feature is enabled, it always reads the entire edit log, computes read length,
corruption length and padding length and shows the following summary
{noformat}
2012-08-27 22:04:38,625 INFO  - Checked the bytes after the end of edit log
  (/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits):
2012-08-27 22:04:38,625 INFO  -   Padding position  = 876 (-1 means padding not found)
2012-08-27 22:04:38,625 INFO  -   Edit log length   = 1065
2012-08-27 22:04:38,625 INFO  -   Read length       = 168
2012-08-27 22:04:38,625 INFO  -   Corruption length = 708
2012-08-27 22:04:38,625 INFO  -   Toleration length = 1024 (= dfs.namenode.edits.toleration.length)
2012-08-27 22:04:38,626 INFO  - Summary: |---------- Read=168 ----------|-- Corrupt=708 --|--
Pad=189 --|
2012-08-27 22:04:38,626 WARN  - Edit log corruption detected:
  corruption length = 708 <= toleration length = 1024; the corruption is tolerable.
{noformat}
- When toleration length is set to 0, it makes sure that there is no corruption in the entire
log, including the padding.  A stray OP_INVALID byte won't be misinterpreted as an end-of-log.
- When toleration length is set to >0, NN starts up only if corruption length <= toleration
length.  If corruption length > toleration length, it throws an exception as below
{noformat}
2012-08-27 22:04:39,123 INFO  - Start checking end of edit log (/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits)
...
2012-08-27 22:04:39,123 DEBUG - found: bytes[0]=0xFF=pad, firstPadPos=169
2012-08-27 22:04:39,123 DEBUG - reset: bytes[1410]=0xAB, pad=0xFF
2012-08-27 22:04:39,124 DEBUG - found: bytes[1411]=0xFF=pad, firstPadPos=1580
2012-08-27 22:04:39,124 INFO  - Checked the bytes after the end of edit log (/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits):
2012-08-27 22:04:39,124 INFO  -   Padding position  = 1580 (-1 means padding not found)
2012-08-27 22:04:39,124 INFO  -   Edit log length   = 2638
2012-08-27 22:04:39,124 INFO  -   Read length       = 169
2012-08-27 22:04:39,124 INFO  -   Corruption length = 1411
2012-08-27 22:04:39,124 INFO  -   Toleration length = 1024 (= dfs.namenode.edits.toleration.length)
2012-08-27 22:04:39,125 INFO  - Summary: |---------- Read=169 ----------|-- Corrupt=1411 --|--
Pad=1058 --|
2012-08-27 22:04:39,125 ERROR - FSNamesystem initialization failed.
java.io.IOException: Edit log corruption detected:
  corruption length = 1411 > toleration length = 1024; the corruption is intolerable.
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkEndOfLog(FSEditLog.java:609)
	...
{noformat}
- Therefore, the recommanded setting is to set the conf to 0 (or a small number).  When corruption
is detected (i.e. NN cannot start up), the corruption length can be read from the log.  Then,
admin could decide whether they are willing to tolerate the corruption or they could try to
recover the edit log by other means.
                
> Further improvement on recovery mode and edit log toleration in branch-1
> ------------------------------------------------------------------------
>
>                 Key: HDFS-3540
>                 URL: https://issues.apache.org/jira/browse/HDFS-3540
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.2.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1.  However, the recovery mode
feature in branch-1 is dramatically different from the recovery mode in trunk since the edit
log implementations in these two branch are different.  For example, there is UNCHECKED_REGION_LENGTH
in branch-1 but not in trunk.
> *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy UNCHECKED_REGION_LENGTH
and to tolerate edit log corruption.
> There are overlaps between these two features.  We study potential further improvement
in this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message