hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5817) mappers get rescheduled on node transition even after all reducers are completed
Date Mon, 10 Aug 2015 22:53:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680903#comment-14680903
] 

Sangjin Lee commented on MAPREDUCE-5817:
----------------------------------------

Thanks for the comments [~bpodgursky]. I've been meaning to revisit this JIRA.

This JIRA is stalled because there are 2 options to move forward with different implications.
Option (1) is to prevent re-running mappers as soon as all reducers complete, and the option
(2) is to add a new task attempt state to be able to handle a situation where a node becomes
unstable while reducers are past the copy phase.

As mentioned above, option (1) is narrower and simpler in scope, but it will ensure that the
job will complete in time only after all reducers are complete. Option (2) can narrow this
window further so that as soon as the we're past the copy phase we can reasonably ensure that
the job will complete in time, but at the expense of a fair amount of complexity of the changes.

FYI, our company has been running with option (1) for a while, and we've been more or less
happy with that.

I'd like to hear others' thoughts on this. The patch can be updated for the option (1) fairly
quickly. It will take more time to complete option (2) on the other hand. We can discuss which
option we want to go with, and even going with option (1) first in this JIRA and do option
(2) in another JIRA. Thoughts?

> mappers get rescheduled on node transition even after all reducers are completed
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5817
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.3.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>              Labels: BB2015-05-TBR
>         Attachments: mapreduce-5817.patch
>
>
> We're seeing a behavior where a job runs long after all reducers were already finished.
We found that the job was rescheduling and running a number of mappers beyond the point of
reducer completion. In one situation, the job ran for some 9 more hours after all reducers
completed!
> This happens because whenever a node transition (to an unusable state) comes into the
app master, it just reschedules all mappers that already ran on the node in all cases.
> Therefore, if any node transition has a potential to extend the job period. Once this
window opens, another node transition can prolong it, and this can happen indefinitely in
theory.
> If there is some instability in the pool (unhealthy, etc.) for a duration, then any big
job is severely vulnerable to this problem.
> If all reducers have been completed, JobImpl.actOnUnusableNode() should not reschedule
mapper tasks. If all reducers are completed, the mapper outputs are no longer needed, and
there is no need to reschedule mapper tasks as they would not be consumed anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message