hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Mon, 05 Apr 2010 17:05:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853441#action_12853441

Suresh Srinivas commented on HDFS-1073:

> The pro of txid numbering is that we can detect the case where some middle log got truncated.
For example, if we have edits_0, edits_1000, and edits_2000, but edits_1000 only contains
500 edits, we can fail at that point

We should generate txid serially. With that it would be easy to determine the missing logs,
based on gaps in txid, even if we decide on naming the files without txids.

> "it decouples the split (ie roll) of the edit log and the checkpoint of the image"
Currently a new edits are rolled when backup node asks the namenode to create one. NN could
have a policy where for every N number of transaction, it could choose to roll edits. Backup
NN could choose to checkpoint fsimage with the relevant edits that have been rolled so far.
This makes rolling of edits a local NN policy and decouples it from backup NN.

> There may be occasional times when we want to roll the edit log when we've not made any
edits into it...
To avoid this kind of complications - if NN periodically rolls edits, checkpointing could
be merging fsimage with all the rolled edits so far. In such a case we need distinction between
edits that are finalized and edit that is being written to.

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message