hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (MAPREDUCE-2205) FairScheduler should not re-schedule jobs that have just been preempted
Date Thu, 16 Dec 2010 19:53:08 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Scott Chen resolved MAPREDUCE-2205.
-----------------------------------

    Resolution: Not A Problem

> FairScheduler should not re-schedule jobs that have just been preempted
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2205
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2205
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>            Reporter: Joydeep Sen Sarma
>            Assignee: Scott Chen
>
> We have hit a problem with the preemption implementation in the FairScheduler where the
following happens:
> # job X runs short of fair share or min share and requests/causes N tasks to be preempted
> # when slots are then scheduled - tasks from some other job are actually scheduled
> # after preemption_interval has passed, job X finds it's still underscheduled and requests
preemption. goto 1.
> This has caused widespread preemption of tasks and the cluster going from high utilization
to low utilization in a few minutes.
> After doing some analysis of the logs - one of the biggest contributing factors seems
to be the scheduling of jobs when a heartbeat with multiple slots is advertised. currently
it goes over all the jobs/pools (in sorted) order until all the slots are exhausted. this
leads to lower priority jobs also getting scheduled (that may have just been preempted).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message