hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Thu, 19 Aug 2010 03:17:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900162#action_12900162

Sanjay Radia commented on HDFS-1073:

Here is what I remember from our meeting in April. Todd, you took notes, please add anything
I missed.
There were 2 issues under contention:
# Add transaction Id to the edit logs
# Name the edit logs and image logs using the transaction id.

These are orthogonal to each other.

Main advantage of adding transaction id to edit logs has following advantages (only the first
advantage was discussed at the meeting, I am adding the other two)
* when a snapshot of a NN state is taken one can record the Tid for the snapshot - this is
useful for knowwing the diff between two snapshots etc.
* while writing  edit logs to multiple files, a failure of the th system can result in different
amounts of data written to each file - the tid allows one to pick one with the most tranasactions.
* In order to do an offline fsck one can needs to dump the block map; clearly one does not
want to the local the system to do an atomic dump. The transaction id of when the dump is
started can be written in the dump to allow the fsck to report consistently.

Main disadvantage is that the editlogs will be little bigger.

Main disadvantage of Naming the edit logs using transaction ids is that the the edit logs
reader needs to be able to seek forward to a specific transaction id. The advantages have
been discussed above; I will summarize in the separate comment.

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs1073.pdf
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message