Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 15 Sep 2016 17:38:21 +0000 (UTC)
From: "Xiaobing Zhou (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13001858.1472687593000.582396.1473961101157@Atlassian.JIRA>
In-Reply-To: <JIRA.13001858.1472687593000@Atlassian.JIRA>
References: <JIRA.13001858.1472687593000@Atlassian.JIRA> <JIRA.13001858.1472687593412@arcas>
Subject: [jira] [Comment Edited] (HDFS-10824)
 MiniDFSCluster#storageCapacities has no effects on real capacity
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 15 Sep 2016 17:38:23 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491523#comment-15491523 ] 

Xiaobing Zhou edited comment on HDFS-10824 at 9/15/16 5:37 PM:
---------------------------------------------------------------

Thanks for review [~anu]. v002 is posted.
1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes is intended for starting additional DNs. so memorizing capacity is changed accordingly. 
2. It's better not to remove storageCapacities parameters, since startDataNodes is designed to start additional DNs in on-going cluster by providing diff capacities.
3. tiggerHeartbeat is to wait for local DN storage to be initialized after block pool has successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll() --> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake, storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although triggerBlock not.
4. passed different capacities.


was (Author: xiaobingo):
Thanks for review [~anu]. v002 is posted.
1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes is intended for starting additional DNs. so memorizing capacity is changed accordingly. 
2. It's better not to remove storageCapacities parameters, since startDataNodes is designed to start additional DNs in on-going cluster by providing diff capacities.
3. tiggerHeartbeat is to wait for for local DN storage to be initialized after block pool has successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll() --> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake, storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although triggerBlock not.
4. passed different capacities.

> MiniDFSCluster#storageCapacities has no effects on real capacity
> ----------------------------------------------------------------
>
>                 Key: HDFS-10824
>                 URL: https://issues.apache.org/jira/browse/HDFS-10824
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, HDFS-10824.002.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real capacity. It can be reproduced by explicitly setting storageCapacities and then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to compare results. The following are  storage report for one node with two volumes after I set capacity as 300 * 1024. Apparently, the capacity is not changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList<E>  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org