hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
Date Tue, 17 Dec 2013 11:02:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850346#comment-13850346

Hudson commented on HDFS-5454:

FAILURE: Integrated in Hadoop-Yarn-trunk #424 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/424/])
HDFS-5454. DataNode UUID should be assigned prior to FsDataset initialization. (Arpit Agarwal)
(arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551296)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeInitStorage.java

> DataNode UUID should be assigned prior to FsDataset initialization
> ------------------------------------------------------------------
>                 Key: HDFS-5454
>                 URL: https://issues.apache.org/jira/browse/HDFS-5454
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Eric Sirianni
>            Assignee: Arpit Agarwal
>            Priority: Minor
>             Fix For: 3.0.0
>         Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch, HDFS-5454.03.patch
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the point where
the {{FsDataset}} object is created ({{DataNode.initStorage()}}.  
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory method, it is
desirable for it to be fully populated with a UUID at this point.  In particular, our {{FsDatasetSpi}}
implementation relies upon the DataNode UUID as a key to access our underlying block storage
> This also appears to be a regression compared to Hadoop 1.x - our 1.x {{FSDatasetInterface}}
plugin has a non-NULL UUID on startup.  I haven't fully traced through the code, but I suspect
this came from the {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.  This greatly
simplifies the fix.  Move the UUID check/generation in from {{DataNode.createBPRegistration()}}
to {{DataNode.initStorage()}}.  This more naturally co-locates UUID generation immediately
subsequent to the read of the UUID from the {{DataStorage}} properties file.
> {code}
>   private void initStorage(final NamespaceInfo nsInfo) throws IOException {
>     // ...
>       final String bpid = nsInfo.getBlockPoolID();
>       //read storage info, lock data dirs and transition fs state if necessary
>       storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>       checkDatanodeUuid();
>     // ...
>   }
> {code}

This message was sent by Atlassian JIRA

View raw message