hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5186) Improve limit handling in fairshare scheduler
Date Mon, 23 Mar 2009 09:16:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688252#action_12688252
] 

Vinod K V commented on HADOOP-5186:
-----------------------------------

One more issue with the current model of limiting by number of jobs per pool is the under-utilization
of map slots while reduce tasks are running. If N is the maximum number of jobs that can run
in a particular pool, when all the maps of first N jobs are done and only the reduces are
running, the map slots sit idle and the maps of next N batch of jobs cannot yet be scheduled.
The fix for this is to change the limits to be in terms of tasks/slots instead of the number
of jobs.

> Improve limit handling in fairshare scheduler
> ---------------------------------------------
>
>                 Key: HADOOP-5186
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5186
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Hemanth Yamijala
>            Priority: Minor
>
> The fairshare scheduler has a way by which it can limit the number of jobs in a pool
by setting the maxRunningJobs parameter in its allocations definition. This limit is treated
as a hard limit, and comes into effect even if the cluster is free to run more jobs, resulting
in underutilization. Possibly the same thing happens with the parameter maxRunningJobs for
user and userMaxJobsDefault. It may help to treat these as a soft limit and run additional
jobs to keep the cluster fully utilized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message