hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
Date Wed, 16 Nov 2011 21:31:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151533#comment-13151533
] 

Todd Lipcon commented on MAPREDUCE-2905:
----------------------------------------

So, I disagree with some of what you said above. The test was broken, but not quite in the
way you described.

The issue is that the test was asserting correct behavior, but the mocking didn't accurately
reflect the way the true scheduler interacts with JobInProgress, etc. In the mocks, as soon
as "obtainNewMapTask" was called, the new task was inserted into the TaskTrackerStatus's taskReports
structure, so that the "countMapTasks" and "countReduceTasks" functions included the just-scheduled
tasks. So, the old code in LoadManager actually did the right thing as far as the test/mock
setup was concerned.

Once we fixed the LoadManager to work correctly with the real code (which doesn't insert anything
into TaskTrackerStatus when the tasks are allocated), it ended up basically double-counting
each assigned task when running against the mocks. So, only half as many tasks were scheduled
as were supposed to.

The fix was to change the mock to obtain all of the scheduled tasks, and only then add them
to the task report structure.

I also had to change the code in the assignment loop to add mapsAssigned and reducesAssigned
around line 476 of FairScheduler.java. Otherwise the "flip flopping" back and forth between
map and reduce task assignment broke.
                
> CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple
per job)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2905
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.20.2
>            Reporter: Jeff Bean
>            Assignee: Jeff Bean
>         Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt,
mr-2905.txt, screenshot-1.jpg
>
>
> We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple,
but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy
just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior
is to spread the job across more nodes so that a relatively small job doesn't saturate any
node in the cluster.
> Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the
desired behavior for small jobs, but is unnecessary for large jobs. However, since this is
a cluster-wide setting, we can't properly tune.
> It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on
submission to better control the task distribution of a particular job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message