hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Wed, 20 Oct 2010 21:00:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923165#action_12923165
] 

Todd Lipcon commented on HDFS-1073:
-----------------------------------

I'm starting to think about how to convert the current code over to the txid based numbering,
came upon a design point I wanted to discuss here:

What should we do about the case when the edits should be rolled, but there have been no transactions
since the last roll? For example, consider the following sequence:
1) Start up a fresh NN. We are writing to file edits_0-inprogress
2) Perform 100 edits- now current txid is 100.
3) Perform a roll. This renames edits_0-inprogress to edits_0-100 and opens edits_100-inprogress
4) No more edits, but a new BN starts up, and thus asks for another roll. Thus we would like
to create edits_100-100, a file with no edits, which is a little bit strange, and will cause
issues the next time we roll (we'll end up with edits_100-100 and also edits_100-200 for example)

It seems the options are:
a) if asked to roll when we have not written any transactions to our current log, it is a
no-op
b) whenever we roll, we append a special "trailer" transaction. Thus every log has at least
1 edit in it. I don't really like this, since it means that after a crash, we'll have a log
without a trailer, which will add edge cases to worry about.

I'm leaning towards A. Am I missing another good solution?

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1073.txt, hdfs1073.pdf
>
>
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message