hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
Date Fri, 05 Sep 2014 04:08:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122378#comment-14122378

James Thomas commented on HDFS-6981:

bq. Not an issue. If there was no layout version change then 'previous' will be absent and
doTransition will skip calling doRollback, instead restoring from trash. An existing test
case already verifies it by starting the DN with the -rollback flag.

Right, I am talking about the case where we roll back to a version of the software prior to
HDFS-6800, in which case the following code will be executed in {{doTransition}}:

    if (startOpt == StartupOption.ROLLBACK) {
      doRollback(sd, nsInfo); // rollback if applicable
    } else {
      // Restore all the files in the trash. The restored files are retained
      // during rolling upgrade rollback. They are deleted during rolling
      // upgrade downgrade.
      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));
      LOG.info("Restored " + restored + " block files from trash.");

So the trash won't be restored until the first heartbeat response is received from the NN,
at which point {{signalRollingUpgrade}} will be called with {{null}}, and we'll call {{restoreTrash}}.
So the blocks in trash will potentially be missing for a short period of time -- not sure
if this is an issue.

> DN upgrade with layout version change should not use trash
> ----------------------------------------------------------
>                 Key: HDFS-6981
>                 URL: https://issues.apache.org/jira/browse/HDFS-6981
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: James Thomas
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, HDFS-6981.03.patch, HDFS-6981.04.patch
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we are running
the old code (prior to this patch), we will restore the previous directory but will not delete
the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the trash
still exists, it will be restored at this point, the appended-to blocks will be overwritten,
and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we have a previous
> Thanks to [~james.thomas] for reporting this.

This message was sent by Atlassian JIRA

View raw message