hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sherry Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2729) Reducers are always counted having "pending tasks" even if they can't be scheduled yet because not enough of their mappers have completed
Date Thu, 04 Aug 2011 16:35:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079451#comment-13079451
] 

Sherry Chen commented on MAPREDUCE-2729:
----------------------------------------

I manually verified this fix on the 10 nodes cluster.

Verification steps:
1. Replace hadoop-capacity-scheduler.jar with the fix on the cluster gateway
2. Modify the capacity-scheduler.xml to ensure a queue have multiple map & reduce task
slots
3. restart mapred
4. Submit jobs for a user which start reduces when 5% (default) maps complete, submit jobs
for 2nd user (same queue as 1st user) which start reduces when 50% maps complete.
5. Verify that 1st user got all queue reduce capacity whatever the 2nd user hasn't used yet,
it is greater than user-limit.



> Reducers are always counted having "pending tasks" even if they can't be scheduled yet
because not enough of their mappers have completed
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2729
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2729
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.205.0
>         Environment: 0.20.1xx-Secondary
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>             Fix For: 0.20.205.0
>
>         Attachments: MAPREDUCE-2729.patch
>
>
> In capacity scheduler, number of users in a queue needing slots are calculated based
on whether users' jobs have any pending tasks.
> This works fine for map tasks. However, for reduce tasks, jobs do not need reduce slots
until the minimum number of map tasks have been completed.
> Here, we add checking whether reduce is ready to schedule (i.e. if a job has completed
enough map tasks) when we increment number of users in a queue needing reduce slots.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message