hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangwj (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
Date Tue, 06 Feb 2018 07:04:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353484#comment-16353484
] 

wangwj commented on YARN-7859:
------------------------------

Hi,[~yufeigu],thank you for your reply.:)
    1、This phenomenon often happens in my cluster because of the large amount of jobs...I
don't think this is a corner cases or particular issue..For example,one queue includes off-line
batch processing job and real time stream processing job..Although this can be solved by disassembling
the queue, but it restricts the flexibility of the queue.
    2、If we add this new property to queue. Few code to use this property. This is different
from maxAMShare.
    3、If we take your suggestion to setting minShare to a smaller value for long-running
job queue.Others queue will must be affected and one or more queues probably can't be shceduled
for a long time. 

> New feature: add queue scheduling deadLine in fairScheduler.
> ------------------------------------------------------------
>
>                 Key: YARN-7859
>                 URL: https://issues.apache.org/jira/browse/YARN-7859
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: wangwj
>            Assignee: wangwj
>            Priority: Major
>              Labels: fairscheduler, features, patch
>             Fix For: 3.0.0
>
>         Attachments: YARN-7859-v1.patch, log, screenshot-1.png, screenshot-3.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>      As everyone knows.In FairScheduler the phenomenon of queue scheduling starvation
often occurs when the number of cluster jobs is large.The App in one or more queue are pending.So
I have thought a way to solve this problem.Add queue scheduling deadLine in fairScheduler.When
a queue is not scheduled for FairScheduler within a specified time.We mandatory scheduler
it!
>     On the basis of the above, I propose this issue...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message