Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 26 Nov 2013 18:34:42 +0000 (UTC)
From: "Eric Sirianni (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12678289.1383936320098.32348.1385490882385@arcas>
In-Reply-To: <JIRA.12678289.1383936320098@arcas>
References: <JIRA.12678289.1383936320098@arcas>
Subject: [jira] [Updated] (HDFS-5484) StorageType and State in
 DatanodeStorageInfo in NameNode is not accurate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Sirianni updated HDFS-5484:
--------------------------------

    Attachment: HDFS-5484-HDFS-2832.patch

Updated patch file addressing review comments.

I spent some time investigating adding a JUnit test as suggested, however this is difficult without more invasive changes.  The {{DatanodeStorage.State}} is basically hardcoded to be {{NORMAL}} in {{FsVolumeImpl}}.  Getting that state to be configurable is fairly invasive (likely involves enhancing {{StorageLocation}} by changing {{dfs.datanode.data.dir}} pattern, etc.).  That's more than I'd like to bite off with this particular patch...

Without having a state other than {{NORMAL}} available, the test is pointless.

> StorageType and State in DatanodeStorageInfo in NameNode is not accurate
> ------------------------------------------------------------------------
>
>                 Key: HDFS-5484
>                 URL: https://issues.apache.org/jira/browse/HDFS-5484
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: Heterogeneous Storage (HDFS-2832)
>            Reporter: Eric Sirianni
>         Attachments: HDFS-5484-HDFS-2832.patch, HDFS-5484-HDFS-2832.patch
>
>
> The fields in DatanodeStorageInfo are updated from two distinct paths:
> # block reports
> # storage reports (via heartbeats)
> The {{state}} and {{storageType}} fields are updated via the Block Report.  However, as seen in the code blow, these fields are populated from a "dummy" {{DatanodeStorage}} object constructed in the DataNode:
> {code}
> BPServiceActor.blockReport() {
> //...
>         // Dummy DatanodeStorage object just for sending the block report.
>         DatanodeStorage dnStorage = new DatanodeStorage(storageID);
> //...
> }
> {code}
> The net effect is that the {{state}} and {{storageType}} fields are always the default of {{NORMAL}} and {{DISK}} in the NameNode.
> The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
> {code}
> public Map<String, BlockListAsLongs> getBlockReports(String bpid);
> {code}
> to:
> {code}
> public Map<DatanodeStorage, BlockListAsLongs> getBlockReports(String bpid);
> {code}
> thereby allowing {{BPServiceActor}} to send the "real" {{DatanodeStorage}} object with the block report.


--
This message was sent by Atlassian JIRA
(v6.1#6144)