hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Wed, 20 Oct 2010 21:00:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923165#action_12923165

Todd Lipcon commented on HDFS-1073:

I'm starting to think about how to convert the current code over to the txid based numbering,
came upon a design point I wanted to discuss here:

What should we do about the case when the edits should be rolled, but there have been no transactions
since the last roll? For example, consider the following sequence:
1) Start up a fresh NN. We are writing to file edits_0-inprogress
2) Perform 100 edits- now current txid is 100.
3) Perform a roll. This renames edits_0-inprogress to edits_0-100 and opens edits_100-inprogress
4) No more edits, but a new BN starts up, and thus asks for another roll. Thus we would like
to create edits_100-100, a file with no edits, which is a little bit strange, and will cause
issues the next time we roll (we'll end up with edits_100-100 and also edits_100-200 for example)

It seems the options are:
a) if asked to roll when we have not written any transactions to our current log, it is a
b) whenever we roll, we append a special "trailer" transaction. Thus every log has at least
1 edit in it. I don't really like this, since it means that after a crash, we'll have a log
without a trailer, which will add edge cases to worry about.

I'm leaning towards A. Am I missing another good solution?

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1073.txt, hdfs1073.pdf
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message