hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vasco (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4613) Scheduling of reduce tasks results in starvation
Date Fri, 31 Aug 2012 16:16:07 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446085#comment-13446085

Vasco commented on MAPREDUCE-4613:

I tried in 2.1.0 (misread 4299 to be resolved in 2.0.1). I can confirm starvation issue is
fixed there. 

With regard to the task scheduling in 2.1.0. If the amount of reducer input is more or less
uniform, then what's the use of scheduling some reduce jobs very early in the job? It don't
think it decreases total job completion time, since the last reduce job can only be scheduled
after all mappers have been scheduled. If there are containers in abundance, then I understand
the use of scheduling reducers as early as possible. But, if containers are scares it sound
more reasonable to me to give all available resources to the mappers.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>                 Key: MAPREDUCE-4613
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 0.23.1, 2.0.1-alpha
>         Environment: 16 (duo core) machine cluster ==> 32 containers
> namenode and resourcemanager running on separate 17th machine
>            Reporter: Vasco
>         Attachments: scheduling.png
> If a job has more reduce tasks than there are containers available, then the reduce tasks
can occupy all containers causing starvation. The attached graph illustrates the behaviour.
Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while
mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption
does not occur.
> A work-around is to set the number of reducers < available containers.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message