hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5654) Not be able to run SLS with FairScheduler
Date Thu, 23 Mar 2017 23:37:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939440#comment-15939440

Yufei Gu commented on YARN-5654:

Uploaded patch v2. 
# Remove class {{ResourceSchedulerWrapper}}. We realize that better to inherit Scheduler Wrapper
from the Real scheduler class instead of AbstractYarnScheduler after an offline discussion
with [~rkanter], so that any public interface changes of FairScheduler could reflect into
the scheduler wrapper. CapacityScheduler did in that way for a long time. The only downside
is we don't automatically have the wrapper for FifoScheduler. 
# There are lots of duplicated code in {{SLSCapacityScheduler}} and {{SLSFairScheduler}}(used
to be in {{ResourceSchedulerWrapper}}). Most of them are related to scheduler metrics. Make
more sense to move them into {{SchedulerMetrics}}. This not only makes extension of SLSFairScheduler
and SLSCapacityScheduler easier,  e.g., right now both SLS scheduler wrapper extend functions
{{allocate()}} and {{handle()}}, we need to extend more functions potentially. This also makes
writing {{SLSFifoScheduler}} easier. File YARN-6383 for FifoScheduler.
# Parameterize unit tests TestAMSimulator and TestNMSimulator so that they can support both
capacity scheduler and fair scheduler. 
# This patch make FairScheduler in SLS work generally. But there are still some bugs to fix,
e.g. the queue metrics and cluster metrics doesn't work well. It is too much to put them in
this jira. I will file followup jira for them.

> Not be able to run SLS with FairScheduler
> -----------------------------------------
>                 Key: YARN-5654
>                 URL: https://issues.apache.org/jira/browse/YARN-5654
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Yufei Gu
>         Attachments: YARN-5654.002.patch, YARN-5654.1.patch
> With the config:
> https://github.com/leftnoteasy/yarn_application_synthesizer/tree/master/configs/hadoop-conf-fs
> And data:
> https://github.com/leftnoteasy/yarn_application_synthesizer/tree/master/data/scheduler-load-test-data
> Capacity Scheduler runs fine, but Fair Scheduler cannot be successfully run. It reports
NPE from RMAppAttemptImpl

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message