hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2201) Quicker preemption causes excessive preemption in FairScheduler
Date Thu, 25 Nov 2010 22:45:15 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935900#action_12935900

Joydeep Sen Sarma commented on MAPREDUCE-2201:

there's something more to it for sure. it seems that after preempting for job X - the freed
up slots were not used for job X - they were used for other jobs (including those just preempted).
i will file a separate jira with details. that seems like bigger issue.

yeah - two heartbeats are required to kill and reschedule. screwed up.

> Quicker preemption causes excessive preemption in FairScheduler
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-2201
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2201
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Joydeep Sen Sarma
> One problem we are seeing is where FairScheduler repeatedly preempts for the same job.
This is presumably because our preemption interval is set to a low number (1 minute). FS queues
up N tasks to be killed - but in 1 min it is not able to kill  and schedule new tasks on all
these slots. As a result, after 1 min - it again preempts a whole bunch of tasks.
> We could (and probably will) workaround this by increasing the preemption interval. However
- this gives us a hard tradeoff between accurate preemption and timely preemption. Not good.
Ideally we want to make the first set of preemptions quickly (to provide responsive behavior
to new jobs for example) - but wait (to make sure that the kill actions have actually been
processed) thereafter.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message