hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1538) Refactor more startup and image loading code out of FSImage
Date Tue, 29 Mar 2011 19:14:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012630#comment-13012630

Ivan Kelly commented on HDFS-1538:

The "VERSION" files are one thing that will need to be discussed/attacked later in this branch.
There are currently some things in there that are image-specific (eg checkpoint time) that
don't really make a ton of sense when multiple images may be present in a directory, and the
image/edit files themselves encode their txids. I will be opening a JIRA to discuss this soon.

The particular reason for this call being distinct from the latestNameSD is that we might
roll edit logs several times without actually saving a checkpoint. This may cause the VERSION
file to get updated in one of the edits directories rather than one of the image directories.

I'm still not 100% sure that this makes sense, though. Can I leave this as is, but add a "TODO"
note to come back to this before merge? We will sweep for TODO/FIXME/etc before the project
is deemed merge-able.
Is your multiple edit directory scenario referring to what may happen post1073? It can happen
now, loadPlan does a sweep to find the latest directory so thats covered, but getStorageDirectoryForProperties()
always returns latestNameSD.

In any case, I've no problem with this becoming a TODO.

Who, then, should set storage.lastCheckpointTxId? To me, this makes sense - storage.lastCheckpointTxId
is the txid
of the last checkpoint (ie fsimage_X) that was either loaded or saved. Hence, the loading
of edit logs should always start at lastCheckpointTxId + 1.
Set it when you load the image or save the image, so that VERSIONS and image are in sync.
This is all fine. What I was suggesting was to, instead of having loadFSEdits retrieve it
from NNStorage, return it from loadFSImage (after setting on nnstorage) and pass it into the
subsequent call to loadFSEdits. I think this would make it clearer that when you load a image
up to point N, you load edits starting from N+1. 

> Refactor more startup and image loading code out of FSImage
> -----------------------------------------------------------
>                 Key: HDFS-1538
>                 URL: https://issues.apache.org/jira/browse/HDFS-1538
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>         Attachments: hdfs-1538-1.txt, hdfs-1538-2.txt, hdfs-1538-on-1521.txt, hdfs-1538-on-branch.2.txt,
hdfs-1538-on-branch.txt, hdfs-1538.txt
> For HDFS-1073, we need to be able to continue to load images in the old "fsimage/edits/edits.new"
layout for the purposes of upgrade.  But that code will be only for backwards compatibility,
and we want to be able to switch to new code for the new layout. This subtask is to separate
out much of that code into an interface which we can implement for both the old and new layouts.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message