hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2202) 0.23: Changes to balancer bandwidth should not require datanode restart.
Date Tue, 26 Jul 2011 16:37:09 GMT
0.23: Changes to balancer bandwidth should not require datanode restart.
------------------------------------------------------------------------

                 Key: HDFS-2202
                 URL: https://issues.apache.org/jira/browse/HDFS-2202
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: balancer, data-node
    Affects Versions: 0.20.205.0
            Reporter: Eric Payne
            Assignee: Eric Payne
             Fix For: 0.20.205.0


Currently in order to change the value of the balancer bandwidth (dfs.datanode.balance.bandwidthPerSec),
the datanode daemon must be restarted.

The optimal value of the bandwidthPerSec parameter is not always (almost never) known at the
time of cluster startup, but only once a new node is placed in the cluster and balancing is
begun. If the balancing is taking too long (bandwidthPerSec is too low) or the balancing is
taking up too much bandwidth (bandwidthPerSec is too high), the cluster must go into a "maintenance
window" where it is unusable while all of the datanodes are bounced. In large clusters of
thousands of nodes, this can be a real maintenance problem because these "mainenance windows"
can take a long time and there may have to be several of them while the bandwidthPerSec is
experimented with and tuned.

A possible solution to this problem would be to add a -bandwidth parameter to the balancer
tool. If bandwidth is supplied, pass the value to the datanodes via the OP_REPLACE_BLOCK and
OP_COPY_BLOCK DataTransferProtocol requests. This would make it necessary, however, to change
the DataTransferProtocol version.





--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message