hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Radwan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
Date Thu, 21 Jun 2012 19:05:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398757#comment-13398757
] 

Ahmed Radwan commented on MAPREDUCE-4358:
-----------------------------------------

@Robert, I think mapreduce.job.reduce.slowstart.completedmaps is related but different. The
issue here is not to wait for a % of mappers to totally complete before start allocating containers
to reducers, but the issue is to prevent reducers from occupying containers while these containers
are still needed by mappers. 
@Jason, watching headroom and preempting reducers should be sufficient to address this issue,
but this doesn't seem to work in our case. It is using Fifo.
@Harsh, MAPREDUCE-4228 seems to address a bug with the behavior of mapreduce.job.reduce.slowstart.completedmaps,
which as I mentioned above is different.
@Sharad, Yes we are seeing this in a customer cluster.
                
> Reducers are assigned containers before all maps are assigned containers
> ------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4358
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.0.0-alpha
>            Reporter: Ahmed Radwan
>
> Reducers start to get containers before all maps are. We have seen this issue and it
is problematic since if there is no avaialable resources for the remaining maps, the job will
just stall where reducers are waiting for mappers which are unable to start because there
is no containers available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message