[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495060#comment-15495060
]
Eric Payne commented on YARN-4945:
----------------------------------
[~sunilg],
I noticed in the resourcemanager log that the metrics were not as I would expect after running
applications. For example, after 1 application has completed running, the {{#queue-active-applications}}
metrics remains 1 instead of 0:
{code}
2016-09-16 01:11:10,189 [SchedulerEventDispatcher:Event Processor] INFO capacity.LeafQueue:
Application removed - appId: application_1473988192446_0001 user: hadoop1 queue: glamdring
#user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0
#queue-active-applications: 1
{code}
After 3 applications have run, the metrics are even more unexpected:
{code}
2016-09-16 01:12:34,622 [SchedulerEventDispatcher:Event Processor] INFO capacity.LeafQueue:
Application removed - appId: application_1473988192446_0003 user: hadoop1 queue: glamdring
#user-pending-applications: -4 #user-active-applications: 4 #queue-pending-applications: 0
#queue-active-applications: 3
{code}
I believe the cause of this is in {{LeafQueue#getAllApplications}}:
{code}
public Collection<FiCaSchedulerApp> getAllApplications() {
Collection<FiCaSchedulerApp> apps =
pendingOrderingPolicy.getSchedulableEntities();
apps.addAll(orderingPolicy.getSchedulableEntities());
return Collections.unmodifiableCollection(apps);
}
{code}
The call to {{pendingOrderingPolicy.getSchedulableEntities()}} returns the {{AbstractComparatorOrderingPolicy#schedulableEntities}}
object, and then the call to {{apps.addAll(orderingPolicy.getSchedulableEntities())}} adds
additional {{FiCaSchedulerApp}}'s to {{schedulableEntities}}.
By creating a copy of the return value of {{pendingOrderingPolicy.getSchedulableEntities()}},
I have been able to verify that the {{schedulableEntities}} does not have extra entries. For
example:
{code}
public Collection<FiCaSchedulerApp> getAllApplications() {
Collection<FiCaSchedulerApp> apps = new TreeSet<FiCaSchedulerApp>(
pendingOrderingPolicy.getSchedulableEntities());
apps.addAll(orderingPolicy.getSchedulableEntities());
return Collections.unmodifiableCollection(apps);
}
{code}
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch,
YARN-2009.v1.patch, YARN-2009.v2.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to support features
like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org
|