hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Tue, 28 Sep 2010 15:24:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915783#action_12915783

Ivan Kelly commented on HDFS-1073:

    while writing edit logs to multiple files, a failure of the th system can result in different
amounts of data written to each file - the tid allows one to pick one with the most tranasactions.

Isn't this also doable by just seeing which as more non-zero bytes? ie seek to the end of
the file, scan backwards through the 0 bytes, and stop. Whichever valid log is longer wins.
Even in the case with the transaction-id, you have to do something like this for a few reasons:
a) we'd rather scan backward from the end of the edit log than forward from the beginning,
since it's going to be a faster startup, and b) even if we see a higher transaction id header
on the last entry, that entry might have been incompletely written to the file, so we still
have to verify that it deserializes correctly.

The case of all edit logs being _inprogress during a crash should be a very rare case. Is
it really an issue if it takes a little longer to determine which has the most transactions
if it's only going to incurred after a bad crash?

I don't think either way has been decided/rejected yet. What you're saying has been my view
- that doing txid based is a bigger change, since we have to introduce the txid concept and
add extra code that allows replaying partial edit log files (ie a subrange of the edits within).
But it's certainly doable and Sanjay has presented some good advantages.

FSEditLog already has a transaction concept which could be modified for this. Currently its
not stored anywhere, but is used for logSync. It starts and 0 at NN startup and increases
monotonically, reseting the next time NN starts.

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs1073.pdf
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message