hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5811) High-memory jobs in CapacityTaskScheduler: Some scenarios make job execution unpredictable
Date Tue, 12 May 2009 09:43:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708379#action_12708379
] 

Vinod K V commented on HADOOP-5811:
-----------------------------------

 - Test I:
  Sumbit 2 normal jobs with the number of map and reduce tasks equalling the cluster capacity
one after another. When one of them starts running and its maps occupies the whole cluster,
submit a high-memory job with the number of tasks equal to the cluster-capacity. After this
job gets submitted, submit 2 more normal jobs. 
  Problem found:
  When maps of the 2 normal jobs finished, the maps of high-memory job were scheduled and
not the reduces of normal second job. Reduces of the second job were blocked for want of memory
till the maps of high-memory job were finished.

 - Test II:
  Submit a high-memory job with number of tasks equal to the cluster-capacity. When this job
starts running, submit 3 normal jobs.
 Problem found:
   Maps of high-memory jobs were scheduled and when the maps got finished, the maps of other
jobs started getting scheduled instead of the reduces of high-memory jobs. Also though the
other jobs were normal, their maps were also not scheduled totally, instead only in one wave.
After one wave, reduce scheduler took over and no task from any job was scheduled till the
tasks of normal jobs started finishing, thus making the scheduling of tasks hard to predict.


> High-memory jobs in CapacityTaskScheduler: Some scenarios make job execution unpredictable
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5811
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5811
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>            Reporter: Vinod K V
>
> Karam found some problems with scheduling for high-memory jobs by CapacityTaskScheduler.
This issue tracks these.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message