hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Fri, 16 Sep 2016 01:26:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495060#comment-15495060
] 

Eric Payne commented on YARN-4945:
----------------------------------

[~sunilg],
I noticed in the resourcemanager log that the metrics were not as I would expect after running
applications. For example, after 1 application has completed running, the {{#queue-active-applications}}
metrics remains 1 instead of 0:
{code}
2016-09-16 01:11:10,189 [SchedulerEventDispatcher:Event Processor] INFO capacity.LeafQueue:
Application removed - appId: application_1473988192446_0001 user: hadoop1 queue: glamdring
#user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0
#queue-active-applications: 1
{code}
After 3 applications have run, the metrics are even more unexpected:
{code}
2016-09-16 01:12:34,622 [SchedulerEventDispatcher:Event Processor] INFO capacity.LeafQueue:
Application removed - appId: application_1473988192446_0003 user: hadoop1 queue: glamdring
#user-pending-applications: -4 #user-active-applications: 4 #queue-pending-applications: 0
#queue-active-applications: 3
{code}
I believe the cause of this is in {{LeafQueue#getAllApplications}}:
{code}
  public Collection<FiCaSchedulerApp> getAllApplications() {
    Collection<FiCaSchedulerApp> apps =
        pendingOrderingPolicy.getSchedulableEntities();
    apps.addAll(orderingPolicy.getSchedulableEntities());

    return Collections.unmodifiableCollection(apps);
  }
{code}
The call to {{pendingOrderingPolicy.getSchedulableEntities()}} returns the {{AbstractComparatorOrderingPolicy#schedulableEntities}}
object, and then the call to {{apps.addAll(orderingPolicy.getSchedulableEntities())}} adds
additional {{FiCaSchedulerApp}}'s to {{schedulableEntities}}.

By creating a copy of the return value of {{pendingOrderingPolicy.getSchedulableEntities()}},
I have been able to verify that the {{schedulableEntities}} does not have extra entries. For
example:
{code}
  public Collection<FiCaSchedulerApp> getAllApplications() {
    Collection<FiCaSchedulerApp> apps = new TreeSet<FiCaSchedulerApp>(
        pendingOrderingPolicy.getSchedulableEntities());
    apps.addAll(orderingPolicy.getSchedulableEntities());

    return Collections.unmodifiableCollection(apps);
  }
{code}

> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch,
YARN-2009.v1.patch, YARN-2009.v2.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to support features
like:
> YARN-2009. YARN-2113. YARN-4781.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message