hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Sirianni (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5505) DataStorage.format() should ignore NamespaceID
Date Tue, 12 Nov 2013 17:42:19 GMT
Eric Sirianni created HDFS-5505:
-----------------------------------

             Summary: DataStorage.format() should ignore NamespaceID
                 Key: HDFS-5505
                 URL: https://issues.apache.org/jira/browse/HDFS-5505
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, federation
    Affects Versions: 2.2.0
            Reporter: Eric Sirianni
            Priority: Minor


I'm observing inconsistencies in the {{DataStorage.namespaceID}} value passed in to my {{FsDatasetSpi}}
plugin:
* Upon _initial_ startup (when the DataNode is formatted), the storage {{DataStorage.namespaceID}}
value is populated with the namespace ID from the NameNode (or an arbitrary namenode in a
federated setup).
* Upon subsequent startups the {{DataStorage.namespaceID}} value is {{NULL}}.

The issue appears to be different handling of the {{DataStorage.namespaceID}} value in {{DataStorage.format()}}
vs. {{DataStorage.setFieldsFromProperties()}}:

{code:title=DataStorage.java}
  void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid) throws IOException
{
    sd.clearDirectory(); // create directory
    this.layoutVersion = HdfsConstants.LAYOUT_VERSION;
    this.clusterID = nsInfo.getClusterID();
    this.namespaceID = nsInfo.getNamespaceID();
  }
....

  protected void setFieldsFromProperties(Properties props, StorageDirectory sd)
    // Read NamespaceID in version before federation
    if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
      setNamespaceID(props, sd);
    }
{code}

If I understand correctly, the proper fix is to leave namespaceID *unset* in {{DataStorage.format()}}.
 The namespaceID here is invalid anyway since it is susceptible to race conditions in federated
NameNode setups -- it will happen to match the namespaceID of the first {{BPOfferService}}
thread that successfully handshakes.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message