hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Mon, 05 Apr 2010 18:26:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853478#action_12853478
] 

Todd Lipcon commented on HDFS-1073:
-----------------------------------

bq. We should generate txid serially. With that it would be easy to determine the missing
logs, based on gaps in txid, even if we decide on naming the files without txids.

But we can only find the gaps by reading the entirety of the files, whereas it's nice to be
able to see the gaps as an operator with a simple "js"

bq. This makes rolling of edits a local NN policy and decouples it from backup NN.

I don't see how this is any different with txid based naming or sequential naming. In either
case we can have the backup/secondary/etc understand how to pull and checkpoint multiple files.
The CheckpointSignature just needs to include the information about which logs have been completed.

bq.  In such a case we need distinction between edits that are finalized and edit that is
being written to.

Yes, I came to this same conclusion this morning when thinking about how to safely restore
failed storage directories.


I should have a preliminary patch this week - so far I have the NN side generally working
(loading, saveNamespace, restart, etc) but haven't started work on the various checkpoint
mechanisms.

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message