hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5186) Improve limit handling in fairshare scheduler
Date Mon, 09 Feb 2009 05:46:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671738#action_12671738

Hemanth Yamijala commented on HADOOP-5186:

Hmm. I am wondering if in that case the unlaunched tasks could get scheduled quicker than
job initialization, which could potentially take a long time, depending on the users code
for setup, or the DFS load etc. It may just be simpler to have an additional 2 or 3 jobs pre-initialized.
I agree it is less optimal than your approach though, but it seems simpler to reason about.

> Improve limit handling in fairshare scheduler
> ---------------------------------------------
>                 Key: HADOOP-5186
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5186
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Hemanth Yamijala
>            Priority: Minor
> The fairshare scheduler has a way by which it can limit the number of jobs in a pool
by setting the maxRunningJobs parameter in its allocations definition. This limit is treated
as a hard limit, and comes into effect even if the cluster is free to run more jobs, resulting
in underutilization. Possibly the same thing happens with the parameter maxRunningJobs for
user and userMaxJobsDefault. It may help to treat these as a soft limit and run additional
jobs to keep the cluster fully utilized.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message