hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Tue, 28 Sep 2010 15:24:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915783#action_12915783
] 

Ivan Kelly commented on HDFS-1073:
----------------------------------

{quote}
    while writing edit logs to multiple files, a failure of the th system can result in different
amounts of data written to each file - the tid allows one to pick one with the most tranasactions.

Isn't this also doable by just seeing which as more non-zero bytes? ie seek to the end of
the file, scan backwards through the 0 bytes, and stop. Whichever valid log is longer wins.
Even in the case with the transaction-id, you have to do something like this for a few reasons:
a) we'd rather scan backward from the end of the edit log than forward from the beginning,
since it's going to be a faster startup, and b) even if we see a higher transaction id header
on the last entry, that entry might have been incompletely written to the file, so we still
have to verify that it deserializes correctly.
{quote}

The case of all edit logs being _inprogress during a crash should be a very rare case. Is
it really an issue if it takes a little longer to determine which has the most transactions
if it's only going to incurred after a bad crash?

{quote}
I don't think either way has been decided/rejected yet. What you're saying has been my view
- that doing txid based is a bigger change, since we have to introduce the txid concept and
add extra code that allows replaying partial edit log files (ie a subrange of the edits within).
But it's certainly doable and Sanjay has presented some good advantages.
{quote}

FSEditLog already has a transaction concept which could be modified for this. Currently its
not stored anywhere, but is used for logSync. It starts and 0 at NN startup and increases
monotonically, reseting the next time NN starts.



> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs1073.pdf
>
>
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message