hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1538) Refactor more startup and image loading code out of FSImage
Date Tue, 29 Mar 2011 15:59:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012534#comment-13012534

Todd Lipcon commented on HDFS-1538:

bq. Why not do latestNameSD.read at the end of doRecovery() and get rid of getStorageDirectoryForProperties()

The "VERSION" files are one thing that will need to be discussed/attacked later in this branch.
There are currently some things in there that are image-specific (eg checkpoint time) that
don't really make a ton of sense when multiple images may be present in a directory, and the
image/edit files themselves encode their txids. I will be opening a JIRA to discuss this soon.

The particular reason for this call being distinct from the latestNameSD is that we might
roll edit logs several times without actually saving a checkpoint. This may cause the VERSION
file to get updated in one of the edits directories rather than one of the image directories.

I'm still not 100% sure that this makes sense, though. Can I leave this as is, but add a "TODO"
note to come back to this before merge? We will sweep for TODO/FIXME/etc before the project
is deemed merge-able.

bq. The previous command takes care of editsVersion!=LAYOUT_VERSION, and imageVersion!=LAYOUT_VERSION
shouldn't cause issues anyhow, as we always have to be able to load from old layout versions

Yes, I think that's my opinion as well - ie that we should feel safe to leave an old version
in place without a re-save. This is similar in logic to the recent comments on HDFS-1780.
I will change this line to a TODO for further discussion as well.

bq. I think it would be clearer for loadFSImage to return the last txid it loaded and then
pass that as a parameter to loadEdits(List<File> editLogs, long fromTxid).

Who, then, should set storage.lastCheckpointTxId? To me, this makes sense - storage.lastCheckpointTxId
is the txid 
of the last checkpoint (ie fsimage_X) that was either loaded or saved. Hence, the loading
of edit logs should always start at lastCheckpointTxId + 1.

bq. Is ImageLoadPlan the best name for this given that it loads Edits also. Why not just LoadPlan?
Will fix.

bq. You should create the logger with FSImageOldStorageInspector so that it's clear which
inspector is logging
Will fix.

> Refactor more startup and image loading code out of FSImage
> -----------------------------------------------------------
>                 Key: HDFS-1538
>                 URL: https://issues.apache.org/jira/browse/HDFS-1538
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>         Attachments: hdfs-1538-1.txt, hdfs-1538-2.txt, hdfs-1538-on-1521.txt, hdfs-1538-on-branch.txt,
> For HDFS-1073, we need to be able to continue to load images in the old "fsimage/edits/edits.new"
layout for the purposes of upgrade.  But that code will be only for backwards compatibility,
and we want to be able to switch to new code for the new layout. This subtask is to separate
out much of that code into an interface which we can implement for both the old and new layouts.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message