hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
Date Wed, 20 Jun 2012 20:00:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397825#comment-13397825
] 

Robert Joseph Evans commented on MAPREDUCE-4358:
------------------------------------------------

Isn't that what mapreduce.job.reduce.slowstart.completedmaps is for?  If you don't want any
reducers to run until all of the maps have finished then you want to set it to 1.0, not the
0.05 that is the default.  I think part of the issue is that the default value for mapreduce.job.reduce.slowstart.completedmaps
is still set for when map and reduce slots were completely separate.  Perhaps this config
does not make since any more now that reduce tasks can block map tasks from running.  Or perhaps
we need another config so that the AM will not fill more than X% of the queue with reduces
until all map tasks have completed.

We have taken the rout of setting the slowstart to 1.0 even on our 1.0.2 clusters because
it improves the cluster utilization and we have not seen much of a hit to the end to end time
of our jobs.
                
> Reducers are assigned containers before all maps are assigned containers
> ------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4358
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.0.0-alpha
>            Reporter: Ahmed Radwan
>
> Reducers start to get containers before all maps are. We have seen this issue and it
is problematic since if there is no avaialable resources for the remaining maps, the job will
just stall where reducers are waiting for mappers which are unable to start because there
is no containers available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message