Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 29 Mar 2011 15:59:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1798664988.19166.1301414345801.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <5029024.144631292444642256.JavaMail.jira@thor>
Subject: [jira] [Commented] (HDFS-1538) Refactor more startup and image
 loading code out of FSImage
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012534#comment-13012534 ] 

Todd Lipcon commented on HDFS-1538:
-----------------------------------

bq. Why not do latestNameSD.read at the end of doRecovery() and get rid of getStorageDirectoryForProperties() altogether?

The "VERSION" files are one thing that will need to be discussed/attacked later in this branch. There are currently some things in there that are image-specific (eg checkpoint time) that don't really make a ton of sense when multiple images may be present in a directory, and the image/edit files themselves encode their txids. I will be opening a JIRA to discuss this soon.

The particular reason for this call being distinct from the latestNameSD is that we might roll edit logs several times without actually saving a checkpoint. This may cause the VERSION file to get updated in one of the edits directories rather than one of the image directories.

I'm still not 100% sure that this makes sense, though. Can I leave this as is, but add a "TODO" note to come back to this before merge? We will sweep for TODO/FIXME/etc before the project is deemed merge-able.

bq. The previous command takes care of editsVersion!=LAYOUT_VERSION, and imageVersion!=LAYOUT_VERSION shouldn't cause issues anyhow, as we always have to be able to load from old layout versions

Yes, I think that's my opinion as well - ie that we should feel safe to leave an old version in place without a re-save. This is similar in logic to the recent comments on HDFS-1780. I will change this line to a TODO for further discussion as well.

bq. I think it would be clearer for loadFSImage to return the last txid it loaded and then pass that as a parameter to loadEdits(List<File> editLogs, long fromTxid).

Who, then, should set storage.lastCheckpointTxId? To me, this makes sense - storage.lastCheckpointTxId is the txid 
of the last checkpoint (ie fsimage_X) that was either loaded or saved. Hence, the loading of edit logs should always start at lastCheckpointTxId + 1.

bq. Is ImageLoadPlan the best name for this given that it loads Edits also. Why not just LoadPlan?
Will fix.

bq. You should create the logger with FSImageOldStorageInspector so that it's clear which inspector is logging
Will fix.

> Refactor more startup and image loading code out of FSImage
> -----------------------------------------------------------
>
>                 Key: HDFS-1538
>                 URL: https://issues.apache.org/jira/browse/HDFS-1538
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: hdfs-1538-1.txt, hdfs-1538-2.txt, hdfs-1538-on-1521.txt, hdfs-1538-on-branch.txt, hdfs-1538.txt
>
>
> For HDFS-1073, we need to be able to continue to load images in the old "fsimage/edits/edits.new" layout for the purposes of upgrade.  But that code will be only for backwards compatibility, and we want to be able to switch to new code for the new layout. This subtask is to separate out much of that code into an interface which we can implement for both the old and new layouts.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira