hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Bean (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
Date Wed, 16 Nov 2011 01:50:51 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150964#comment-13150964
] 

Jeff Bean commented on MAPREDUCE-2905:
--------------------------------------

testFairScheduler is buggy. In one instance it actually looks for the bug condition and passes
it by asserting that tt1 fills to capacity on small jobs that shouldn't clump together:

    checkAssignment("tt1", "attempt_test_0001_m_000000_0 on tt1",
                           "attempt_test_0001_r_000000_0 on tt1",
                           "attempt_test_0002_m_000000_0 on tt1",
                           "attempt_test_0002_r_000000_0 on tt1");
    checkAssignment("tt2", "attempt_test_0001_m_000001_0 on tt2",
                           "attempt_test_0002_r_000001_0 on tt2");

If tt1 has a capacity of 2 mappers and 2 reducers, I expect 1 mapper and 1 reducer to be assigned
each as to assign all 4 slots would be the clumping condition that this patch is trying to
address.

On every instance of checkAssignment that gets multiple tasks, the test fails on the length
of the task list. The assignMultiple tests are broken for the reason I describe above. The
other tests that fail are the testDelaySchedulingAtNodeLevel, testDelaySchedulingAtRackLevel,
and testDelaySchedulingOffRack. In all three of those cases a task list length of 2 of the
same type of is expected and a task list length of 1 is returned. Again, with a default capacity
of 2 slots per task type, we'd be over capacity and actually looking for the condition this
patch fixes.

Relevant output from failure:

Testcase: testSmallJobsWithAssignMultiple took 0.58 sec
        FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
        at org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
        at org.apache.hadoop.mapred.TestFairScheduler.testSmallJobsWithAssignMultiple(TestFairScheduler.java:784)

Testcase: testLargeJobs took 0.432 sec
Testcase: testLargeJobsWithAssignMultiple took 0.435 sec
        FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
        at org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
        at org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithAssignMultiple(TestFairScheduler.java:954)

Testcase: testJobsWithPriorities took 0.432 sec
Testcase: testLargeJobsWithPools took 0.672 sec
Testcase: testLargeJobsWithExcessCapacity took 0.598 sec
Testcase: testLargeJobsWithExcessCapacityAndAssignMultiple took 0.594 sec
        FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
        at org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
        at org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithExcessCapacityAndAssignMultiple(TestFairScheduler.java:1301)

Testcase: testSmallJobInLargePool took 0.517 sec
Testcase: testPoolMaxJobs took 0.842 sec
Testcase: testUserMaxJobs took 0.73 sec
Testcase: testComplexJobLimits took 2.053 sec
Testcase: testSizeBasedWeight took 0.387 sec
Testcase: testPoolWeights took 0.851 sec
Testcase: testPoolWeightsWhenNoMaps took 0.637 sec
Testcase: testPoolMaxMapsReduces took 0.404 sec
Testcase: testCapBasedLoadManager took 0.119 sec
Testcase: testMinSharePreemption took 0.487 sec
Testcase: testMinSharePreemptionWithSmallJob took 0.426 sec
Testcase: testFairSharePreemption took 0.612 sec
Testcase: testFairSharePreemptionFromMultiplePools took 0.602 sec
Testcase: testMinAndFairSharePreemption took 0.42 sec
Testcase: testNoPreemptionIfDisabled took 0.457 sec
Testcase: testNoPreemptionIfOnlyLogging took 0.458 sec
Testcase: testDelaySchedulingAtNodeLevel took 0.366 sec
        FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
        at org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
        at org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtNodeLevel(TestFairScheduler.java:2301)

Testcase: testDelaySchedulingAtRackLevel took 0.521 sec
        FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
        at org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
        at org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtRackLevel(TestFairScheduler.java:2349)

Testcase: testDelaySchedulingOffRack took 0.261 sec
        FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>


This is every time checkAssignment() gets called with more than one task as a parameter. This
is a buggy test. For example, in one instance it actuall 

                
> CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple
per job)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2905
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.20.2
>            Reporter: Jeff Bean
>            Assignee: Jeff Bean
>         Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt,
screenshot-1.jpg
>
>
> We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple,
but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy
just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior
is to spread the job across more nodes so that a relatively small job doesn't saturate any
node in the cluster.
> Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the
desired behavior for small jobs, but is unnecessary for large jobs. However, since this is
a cluster-wide setting, we can't properly tune.
> It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on
submission to better control the task distribution of a particular job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message