hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7466) Allow different values for dfs.datanode.balance.max.concurrent.moves per datanode
Date Mon, 14 Sep 2015 18:47:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744012#comment-14744012

Daniel Templeton commented on HDFS-7466:

[~benoy], are you still working on this?

Here's a wacky idea.  Since the DNs will return throttling errors if the thread limit is exceeded,
what if the balancer/mover launches threads until it hits a throttle error and takes that
as the empirical limit?  That approach also has the advantage of reacting to the real load,
i.e. if there are other things tying up the DN's threads then the balancer will reduce it's
number of sending threads accordingly.

> Allow different values for dfs.datanode.balance.max.concurrent.moves per datanode
> ---------------------------------------------------------------------------------
>                 Key: HDFS-7466
>                 URL: https://issues.apache.org/jira/browse/HDFS-7466
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>    Affects Versions: 2.6.0
>            Reporter: Benoy Antony
>            Assignee: Benoy Antony
> It is possible to configure different values for  _dfs.datanode.balance.max.concurrent.moves_
per datanode.  But the value will be used by balancer/mover which obtains the value from its
own configuration. 
> The correct approach will be to obtain the value from the datanode itself.

This message was sent by Atlassian JIRA

View raw message