Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 20 Nov 2017 09:11:01 +0000 (UTC)
From: "Gang Xie (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13118771.1510797357000.278564.1511169061271@Atlassian.JIRA>
In-Reply-To: <JIRA.13118771.1510797357000@Atlassian.JIRA>
References: <JIRA.13118771.1510797357000@Atlassian.JIRA> <JIRA.13118771.1510797357718@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-12820) Decommissioned datanode is counted
 in service cause datanode allcating failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 20 Nov 2017 09:11:08 -0000


    [ https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258975#comment-16258975 ] 

Gang Xie commented on HDFS-12820:
---------------------------------

And I believe this issue still in the latest version

> Decommissioned datanode is counted in service cause datanode allcating failure
> ------------------------------------------------------------------------------
>
>                 Key: HDFS-12820
>                 URL: https://issues.apache.org/jira/browse/HDFS-12820
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: block placement
>    Affects Versions: 2.4.0
>            Reporter: Gang Xie
>
> When allocate a datanode when dfsclient write with considering the load, it checks if the datanode is overloaded by calculating the average xceivers of all the in service datanode. But if the datanode is decommissioned and become dead, it's still treated as in service, which make the average load much more than the real one especially when the number of the decommissioned datanode is great. In our cluster, 180 datanode, and 100 of them decommissioned, and the average load is 17. This failed all the datanode allocation. 
> private void subtract(final DatanodeDescriptor node) {
>       capacityUsed -= node.getDfsUsed();
>       blockPoolUsed -= node.getBlockPoolUsed();
>       xceiverCount -= node.getXceiverCount();
>     {color:red}  if (!(node.isDecommissionInProgress() || node.isDecommissioned())) {{color}
>         nodesInService--;
>         nodesInServiceXceiverCount -= node.getXceiverCount();
>         capacityTotal -= node.getCapacity();
>         capacityRemaining -= node.getRemaining();
>       } else {
>         capacityTotal -= node.getDfsUsed();
>       }
>       cacheCapacity -= node.getCacheCapacity();
>       cacheUsed -= node.getCacheUsed();
>     }


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org