hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Rewoonenco (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7152) add command-line or configuration options for balancer (tweak speed)
Date Fri, 26 Sep 2014 15:22:34 GMT
Andrew Rewoonenco created HDFS-7152:

             Summary: add command-line or configuration options for balancer (tweak speed)
                 Key: HDFS-7152
                 URL: https://issues.apache.org/jira/browse/HDFS-7152
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: balancer
    Affects Versions: 2.5.0, 2.3.0, 2.6.0
            Reporter: Andrew Rewoonenco

Make a command-line or configuration options for balancer (hints to process it work faster):

1. Add option to filter minimal and maximal block size.
 a) when datanode holds a lot of small files and a couple of big ones balancer do senseless
balancing on small files, it take a lot of time to nothing.
 b) when datanode have a large and very large files balancer sometimes stuck on moving very
large files fails with timeouts.
 So it is good to limit such actions.

2. Add option for block move timeout and iteration timeout.
  - in version 2.3.0 - 2.5.0 socket use non-configurable socket timeout of 60 seconds making
use of balancer is useless when block size of HDFS greater 2 Gb.
  - in version 2.6.0 and later hard-coded values used for balancer iteration time.
  They need to be replaced by configurable ones.

This message was sent by Atlassian JIRA

View raw message