hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
Date Mon, 05 Feb 2018 20:49:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352921#comment-16352921
] 

Yufei Gu commented on YARN-7859:
--------------------------------

Basically, we should be very careful about introducing new properties to queue. Take {{maxAMShare}}
as an example. Community spends years to make it right. There are always corner cases because
of combination of multiple queue properties. I am not sure whether it is worth to introduce
one queue property to solve one corner case which potentially introduce more corner cases.


Global scheduling is the right way to solve this kind of issue and on the todo list of community.
We could postpone this or thinking about alternatives. For this particular issue, there is
one easy way to mitigate, which is setting minShare to a smaller value for long-running job
queue. In that case, its minShare usages are supposed to be larger and get more chance to
be scheduled. 

[~miklos.szegedi@cloudera.com], [~wilfreds], what's your take on this?

> New feature: add queue scheduling deadLine in fairScheduler.
> ------------------------------------------------------------
>
>                 Key: YARN-7859
>                 URL: https://issues.apache.org/jira/browse/YARN-7859
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: wangwj
>            Assignee: wangwj
>            Priority: Major
>              Labels: fairscheduler, features, patch
>             Fix For: 3.0.0
>
>         Attachments: YARN-7859-v1.patch, log, screenshot-1.png, screenshot-3.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>      As everyone knows.In FairScheduler the phenomenon of queue scheduling starvation
often occurs when the number of cluster jobs is large.The App in one or more queue are pending.So
I have thought a way to solve this problem.Add queue scheduling deadLine in fairScheduler.When
a queue is not scheduled for FairScheduler within a specified time.We mandatory scheduler
it!
>     On the basis of the above, I propose this issue...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message