hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaobing Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity
Date Thu, 15 Sep 2016 17:38:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491523#comment-15491523
] 

Xiaobing Zhou edited comment on HDFS-10824 at 9/15/16 5:37 PM:
---------------------------------------------------------------

Thanks for review [~anu]. v002 is posted.
1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes
is intended for starting additional DNs. so memorizing capacity is changed accordingly. 
2. It's better not to remove storageCapacities parameters, since startDataNodes is designed
to start additional DNs in on-going cluster by providing diff capacities.
3. tiggerHeartbeat is to wait for local DN storage to be initialized after block pool has
successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll()
--> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake,
storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although
triggerBlock not.
4. passed different capacities.


was (Author: xiaobingo):
Thanks for review [~anu]. v002 is posted.
1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes
is intended for starting additional DNs. so memorizing capacity is changed accordingly. 
2. It's better not to remove storageCapacities parameters, since startDataNodes is designed
to start additional DNs in on-going cluster by providing diff capacities.
3. tiggerHeartbeat is to wait for for local DN storage to be initialized after block pool
has successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll()
--> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake,
storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although
triggerBlock not.
4. passed different capacities.

> MiniDFSCluster#storageCapacities has no effects on real capacity
> ----------------------------------------------------------------
>
>                 Key: HDFS-10824
>                 URL: https://issues.apache.org/jira/browse/HDFS-10824
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, HDFS-10824.002.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real capacity.
It can be reproduced by explicitly setting storageCapacities and then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE)
to compare results. The following are  storage report for one node with two volumes after
I set capacity as 300 * 1024. Apparently, the capacity is not changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList<E>  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message