hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
Date Wed, 04 Dec 2013 23:36:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13839483#comment-13839483

Carlo Curino commented on YARN-1471:

The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific
to the CapacityScheduler.

The three offending methods are: 
* scheduler.getResourceCalculator();
* scheduler.getRootQueue(); (which is a CSQueue)
* scheduler.getClusterResources()

However more generally this policy only makes sense when paired with the CapacityScheduler,
so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler.

This is coming up again in the work towards YARN-1051, as we have other "monitors" interacting
with the underlying scheduler, possibly in ways that are specific to one specific scheduler.

Speaking with [~chris.douglas] we seem to agree that the simulator would be much more useful
if it allows to simulate specifics of each scheduler more than just exercise the general API
(specifically I would not use it otherwise), and hacking the Scheduler API (and classes using
it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems
a job for aspect programming... though it comes with its own set of headaches (see mumak).

Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler
to those (assuming the rest of your metrics are collected in methods not used by the monitor).

> The SLS simulator is not running the preemption policy for CapacityScheduler
> ----------------------------------------------------------------------------
>                 Key: YARN-1471
>                 URL: https://issues.apache.org/jira/browse/YARN-1471
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Carlo Curino
>            Assignee: Wei Yan
>            Priority: Minor
> The simulator does not run the ProportionalCapacityPreemptionPolicy monitor.  This is
because the policy needs to interact with a CapacityScheduler, and the wrapping done by the
simulator breaks this. 

This message was sent by Atlassian JIRA

View raw message