hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky
Date Thu, 25 May 2017 20:09:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025306#comment-16025306

Lei (Eddy) Xu commented on HDFS-11682:

[~manojg], I should make it clearer, the problem is that there might not HB/BR between two
_iterations_ in {{Balancer#runOneIteration}}. So that before we run {{waitForBalancer}}, the
balancer task may have already finished, as the Balancer thought the avg utilization is {{40}}.
As mentioned in the previous comments, the actual average utilization is {{38.7}}, so that
the utilization of that particular DN {{50}} is just about larger than {{avgUtilization +
threshold + delta (38.7 + 10 + 1)}} . The fix here is that, making sure that {{Balancer}}
can run after another HB/BR with an accurate utilization number.  

[~andrew.wang], the miscalculation is due to the out-of-date NN blocks between _two iterations_
of {{Balancer#runOneIteration}}. So it is hard to trigger HB/BR with in the {{Balancer}} itself.
 Alternatively, how about make {{timeout}} in {{TestBalancer#waitForBalancer}} shorter, i.e.,
10s? So the execution time in the worst case is about the same.  


> TestBalancer#testBalancerWithStripedFile is flaky
> -------------------------------------------------
>                 Key: HDFS-11682
>                 URL: https://issues.apache.org/jira/browse/HDFS-11682
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Andrew Wang
>            Assignee: Lei (Eddy) Xu
>         Attachments: HDFS-11682.00.patch, HDFS-11682.01.patch, IndexOutOfBoundsException.log,
> Saw this fail in two different ways on a precommit run, but pass locally.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message