hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1030) Reduce tasks are getting starved in capacity scheduler
Date Mon, 05 Oct 2009 05:27:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762091#action_12762091

Hemanth Yamijala commented on MAPREDUCE-1030:

Some comments, all in test cases:
- The intent of the test case, testMaxReduceCap, is slightly changed. Previously, we wanted
to verify that after hitting the max reduce limit, finishing a map task has no effect. This
check is missing.
- testCapacityTransfer - can check that both maps and reduces are being assigned.
- testHighMemoryBlockingWithMaxLimit can verify that all tasks are assigned as expected, and
not just check for maps or reduces.
- For the testcase, testUserLimitsForHighMemoryJobs, I think we can make it more readable
if we put it in a loop, as it is repeating the same thing. Something like:
for (int i=0; i<5; i++) {
      CapacityTestUtils.MAP, "attempt_test_0001_m_00000"+(i+1)+"_0 on tt1");
      CapacityTestUtils.REDUCE, "attempt_test_0001_r_00000"+(i+1)+"_0 on tt1");
  checkAssignmentTasks(taskTrackerManager, scheduler, "tt1",
This will also avoid mistakes like for instance in one of the scenarios here we should be
using checkAssignmentTasks but instead we are using checkAssignment
- In testSchedulingInformation, check for reduce stats being updated after job3 is scheduled.
- testHighMemoryBlockingAcrossTaskTypes - after the first call to assignTasks, a map and reduce
are both scheduled. We should have another call that will now assert that a reduce of the
second job is scheduled.
- testSpeculativeTaskScheduling: The assertion for fjob1.pendingReduces() to be 0 can be moved
after the initial call to assignTasks itself with multiple task assignment. Also, we are not
checking assignment of speculative reduces. After speculative tasks are handled, we should
check that the normal job's reduces are also scheduled.
- testMultiTaskAssignmentInSingleQueue() is no longer needed, as the same is being tested
in many other test cases.
- testMultiTaskAssignmentInMultipleQueues() needs a javadoc for explaining the test case,
and should reuse the new utility APIs like checkAssignmentTasks().
- checkRunningJobMovementAndCompletion can check multiple assignment.
- checkAssignment - indentation seems wrong.
- Since checkAssignment cannot be called if no tasks are returned, the check for tasks ==
null or empty seems unnecessary. Also, we can simply return the 0th element in the list of
- We can rename checkAssignmentTasks to checkMultipleTaskAssignment() to better document intent.
This also needs a javadoc. Also, indentation within the method needs some correction. Also,
I think we should also check that the number of tasks assigned matches the number expected.
- TestContainerQueue and TestRefreshOfQueues test classes can be easily modified for checking
multiple assignment because of the utility APIs we have. This way, we can actually check reduce
assignment for hierarchical queues is working as expected as well.

> Reduce tasks are getting starved in capacity scheduler
> ------------------------------------------------------
>                 Key: MAPREDUCE-1030
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1030
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.21.0
>            Reporter: rahul k singh
>            Assignee: rahul k singh
>            Priority: Blocker
>             Fix For: 0.21.0
>         Attachments: MAPREDUCE-1030-1.patch, MAPREDUCE-1030-2.patch.txt, MAPREDUCE-1030-3.patch,
MAPREDUCE-1030-4.patch, MAPREDUCE-1030-5.patch
> reduce tasks are getting starved in capacity scheduler. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message