hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9730) Storage ID update does not happen when there is a layout change
Date Tue, 02 Feb 2016 03:47:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127613#comment-15127613
] 

Vinayakumar B commented on HDFS-9730:
-------------------------------------

Thanks [~kihwal] for reporting this. Yes, indeed {{createStorageId(..)}} is not called during
layout upgrade.
bq. We still call createStorageID(..) the upgrade path. The call is moved to doUgrade(..)
as shown below (line 802 in DataStorage).
Yes [~szetszwo], its called. But it will hit only during layout upgrade from pre-federation
layouts. Otherwise it will just write latest layoutVersion to VERSION file and return. does
nothing else.
{code}    if (DataNodeLayoutVersion.supports(
        LayoutVersion.Feature.FEDERATION, layoutVersion)) {
      // The VERSION file is already read in. Override the layoutVersion 
      // field and overwrite the file. The upgrade work is handled by
      // {@link BlockPoolSliceStorage#doUpgrade}
      LOG.info("Updating layout version from " + layoutVersion + " to "
          + HdfsServerConstants.DATANODE_LAYOUT_VERSION + " for storage "
          + sd.getRoot());
      layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION;
      writeProperties(sd);
      return;
    }{code}

Change created this is 
{code}     // do upgrade
     if (this.layoutVersion > HdfsServerConstants.DATANODE_LAYOUT_VERSION) {
-      doUpgrade(datanode, sd, nsInfo);  // upgrade
-      createStorageID(sd, !haveValidStorageId);
-      return;
+      doUpgrade(sd, nsInfo, conf);  // upgrade
+      return true; // doUgrade already has written properties
     }{code}

My Apologies for missing this during review :)

The fix would be as below.
{code}--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
@@ -743,6 +743,8 @@ void doUpgrade(final StorageDirectory sd, final NamespaceInfo nsInfo,
       LOG.info("Updating layout version from " + layoutVersion + " to "
           + HdfsServerConstants.DATANODE_LAYOUT_VERSION + " for storage "
           + sd.getRoot());
+      // check and regenerate storageId if required.
+      createStorageID(sd, layoutVersion);
       layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION;
       writeProperties(sd);
       return;
{code}

But, I have tried to run tests by changing layout version in my eclipse, didnt hit failures
though ;)


> Storage ID update does not happen when there is a layout change
> ---------------------------------------------------------------
>
>                 Key: HDFS-9730
>                 URL: https://issues.apache.org/jira/browse/HDFS-9730
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Assignee: Tsz Wo Nicholas Sze
>
> HDFS-9654 will cause test failures when we increment the datanode layout version next
time.
> {noformat}
> TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs
> TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22FixesStorageIDs
> {noformat}
> This is because createStorageID() is no longer called when it goes through the layout
upgrade path. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message