hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4560) Job can get stuck in a deadlock between mappers and reducers for low values of mapreduce.job.reduce.slowstart.completedmaps (<<1)
Date Thu, 04 Oct 2012 17:03:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469511#comment-13469511
] 

Rahul Jain commented on MAPREDUCE-4560:
---------------------------------------

Yes, this issue was found in FIFO scheduler; we can mark it duplicate of MAPREDUCE-4299 once
we verify that fix does resolve the issue.
                
> Job can get stuck in a deadlock between mappers and reducers for low values of mapreduce.job.reduce.slowstart.completedmaps
(<<1)
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4560
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4560
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Rahul Jain
>             Fix For: 2.0.0-alpha
>
>
> This issue has been seen with MapReduceV2, never with MapReduceV1 in our lab systems.
> The parameter mapreduce.job.reduce.slowstart.completedmaps=0.05 (the default value).
> We found Application master stuck in a deadlock between mappers and reducers with no
progress in the job; the sequence appears to be:
> 1. Initial available map/reduce slots were allocated to mappers
> 2. Once mappers made progress and few of them completed, reducers started occupying few
of the slots due to low values of above config param.
> 3. The scheduler appears to not give priority to mappers over reducers; after a while
in our system we saw all slots occupied by reducers.
> 4. Since there were still mapper tasks not yet assigned any slot, the map phase never
completed.
> 5. The system entered a deadlock state where reducers occupy all available slots, but
are waiting for mappers to be complete; mappers cannot move forward because of no slot available.
> The workaround in our system was to set 
> mapreduce.job.reduce.slowstart.completedmaps=1 and the issue was no longer seen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message