hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
Date Thu, 22 Feb 2018 02:30:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372338#comment-16372338
] 

Yufei Gu commented on YARN-7859:
--------------------------------

Hi [~wangwj], let me reiterate my stands. This is a real issue needs to be solve. Thank you
for filing this. I am open to any proposal. Unfortunately the solution you propose here breaks
the fairness, which leads to potential issues. A new queue property could also bring lots
of corner cases. Other than that, global scheduling seems a right way to fix this issue. You
can either wait for it or work on it. Before global scheduling is available, there are some
workarounds(decreasing minShare of long-running job) to mitigate. Global scheduling may not
be the only right solution. I would also recommend you rethink the solution, so that it doesn't
break fairness, and validate potential risks thoroughly if you want to introduce any new queue
property. Anyway, thank you for filing this. Great job so far. 

> New feature: add queue scheduling deadLine in fairScheduler.
> ------------------------------------------------------------
>
>                 Key: YARN-7859
>                 URL: https://issues.apache.org/jira/browse/YARN-7859
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: wangwj
>            Assignee: wangwj
>            Priority: Major
>              Labels: fairscheduler, features, patch
>             Fix For: 3.0.0
>
>         Attachments: YARN-7859-v1.patch, YARN-7859-v2.patch, log, screenshot-1.png, screenshot-3.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>      As everyone knows.In FairScheduler the phenomenon of queue scheduling starvation
often occurs when the number of cluster jobs is large.The App in one or more queue are pending.So
I have thought a way to solve this problem.Add queue scheduling deadLine in fairScheduler.When
a queue is not scheduled for FairScheduler within a specified time.We mandatory scheduler
it!
>     On the basis of the above, I propose this issue...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message