hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Sirianni (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5505) DataStorage.format() should ignore NamespaceID
Date Tue, 12 Nov 2013 17:42:19 GMT
Eric Sirianni created HDFS-5505:

             Summary: DataStorage.format() should ignore NamespaceID
                 Key: HDFS-5505
                 URL: https://issues.apache.org/jira/browse/HDFS-5505
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, federation
    Affects Versions: 2.2.0
            Reporter: Eric Sirianni
            Priority: Minor

I'm observing inconsistencies in the {{DataStorage.namespaceID}} value passed in to my {{FsDatasetSpi}}
* Upon _initial_ startup (when the DataNode is formatted), the storage {{DataStorage.namespaceID}}
value is populated with the namespace ID from the NameNode (or an arbitrary namenode in a
federated setup).
* Upon subsequent startups the {{DataStorage.namespaceID}} value is {{NULL}}.

The issue appears to be different handling of the {{DataStorage.namespaceID}} value in {{DataStorage.format()}}
vs. {{DataStorage.setFieldsFromProperties()}}:

  void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid) throws IOException
    sd.clearDirectory(); // create directory
    this.layoutVersion = HdfsConstants.LAYOUT_VERSION;
    this.clusterID = nsInfo.getClusterID();
    this.namespaceID = nsInfo.getNamespaceID();

  protected void setFieldsFromProperties(Properties props, StorageDirectory sd)
    // Read NamespaceID in version before federation
    if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
      setNamespaceID(props, sd);

If I understand correctly, the proper fix is to leave namespaceID *unset* in {{DataStorage.format()}}.
 The namespaceID here is invalid anyway since it is susceptible to race conditions in federated
NameNode setups -- it will happen to match the namespaceID of the first {{BPOfferService}}
thread that successfully handshakes.

This message was sent by Atlassian JIRA

View raw message