hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen He (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-6159) TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if there is block missing after balancer success
Date Wed, 02 Apr 2014 23:35:19 GMT

     [ https://issues.apache.org/jira/browse/HDFS-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chen He updated HDFS-6159:
--------------------------

    Attachment: HDFS-6159-v2.patch

The balancer will create a balancer.id file (about 10B in sieze) in /system (HDFS) directory
when it starts and delete it after it is done. However, this balancer.id file affects cluster's
average utilization if a cluster is made of datanodes with limited capacities. In the TestBalancerWithNodeGroup
class, each datanode only has 500B total capacity for HDFS and the data block size is 10B.
I propose to increase datanode capacity up to 6000B and data block size to 100B.

 Here are the reasons:
1) Increasing datanode capacity can reduce the fluctuation caused by the balancer.id file;
2) To avoid increasing the number of blocks that the balancer needs to move, I increase the
block size accordingly.

Patch is available for reviewing.

> TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if there is block missing after
balancer success
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6159
>                 URL: https://issues.apache.org/jira/browse/HDFS-6159
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Chen He
>            Assignee: Chen He
>         Attachments: HDFS-6159-v2.patch, HDFS-6159.patch, logs.txt
>
>
> The TestBalancerWithNodeGroup.testBalancerWithNodeGroup will report negative false failure
if there is(are) data block(s) losing after balancer successfuly finishes. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message