hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Chansler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Sat, 23 Oct 2010 01:03:23 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924103#action_12924103

Robert Chansler commented on HDFS-1073:

Is the file 100-100 _forbidden_? What if the service is stopped when the most recent file
has zero records? (I'd always write a "I'm quitting" record, otherwise you can never know
if you have lost the last edits.) And what if there are files 100-200 and 100-300? Rather
than different special cases, why not make the general case just work? Roll means roll regardless,
and starting up finds the latest image and _any_ consistent sequence of edits that -starts
with- includes the very next transaction, reporting whether the last available edit record
is "I'm quitting!".

And catching up with Sanjay's comment about tx ids in every record, it would seem that the
principal benefits are really obtained only if the tx id is assigned to requests as they are
_received in sequence_. Just doing {{log.write(id++)}} doesn't offer much real protection.

If there is a tx id per record, would it make sense for the actual bits be the record check
sum+id? Years ago we discussed having record check sums, but it never became a priority.

(In file N-M, I might have expected that the first record, if any, has tx id N, not N+1.)

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1073.txt, hdfs1073.pdf
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message