hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
Date Thu, 22 Sep 2011 16:21:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112686#comment-13112686

Robert Joseph Evans commented on MAPREDUCE-3050:

bq. SchedulerAppImpl.java: IIUC, getReservedContainers() is new. It seems to be very similar
to getAllReservedContainers(). Do we need both, and if so, how are they different?

getAllReservedContainers is synchronized properly where getReservedContainers is not :). 
Also getReservedContainers is a public method exposed through the interface, so I will make
just one method.

bq. YarnScheduler.java: getApplication() is @Private

@Private is a thing that Hadoop invented and is not a standard java thing.  This is just saying
that even though the method is public according to java it is not intended to be used by other
components.  About the only thing it does is remove it's documentation from javadocs.  I suppose
LimitedPrivate("yarn") might be better, but I marked it @Private because Queue is @Private
and I did not want to expose a @Private class from a @Public method.

bq. getRootQueue(): The implementation of this method returns different types from CapacityScheduler
vs. FifoScheduler.

The YarnScheduler interface returns a Queue.  CapacityScheduler and FifoScheduler each return
an implementation of Queue.  What this means is that the following code will compile.

YarnScheduler sched = ...;
Queue q = sched.getRootQueue();
if(sched instanceof CapcityScheduler) {
    CSQueue cq = ((CapacityScheduler)sched).getRootQueue();

> YarnScheduler needs to expose Resource Usage Information
> --------------------------------------------------------
>                 Key: MAPREDUCE-3050
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>            Priority: Blocker
>             Fix For: 0.23.0, 0.24.0
>         Attachments: MR-3050.txt
> Before the recent refactor The nodes had information in them about how much resources
they were using.  This information is not hidden inside SchedulerNode.  Similarly resource
usage information about an application, or in aggregate is only available through the Scheduler
and there is not interface to pull it out.
> We need to expose APIs to get Resource and Container information from the scheduler,
in aggregate across the entire cluster, per application, per node, and ideally also per queue
if applicable (although there are no JIRAs I am aware of that need this right now).

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message