hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4690) Skip object allocation in FSAppAttempt#getResourceUsage when possible
Date Thu, 18 Feb 2016 04:24:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151727#comment-15151727
] 

Sangjin Lee commented on YARN-4690:
-----------------------------------

Sounds good. Hadn't checked what {{getPreemptedResources()}} does.

I'm also +1. I'll commit it shortly.

> Skip object allocation in FSAppAttempt#getResourceUsage when possible
> ---------------------------------------------------------------------
>
>                 Key: YARN-4690
>                 URL: https://issues.apache.org/jira/browse/YARN-4690
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: YARN-4690.patch
>
>
> YARN-2768 addresses an important bottleneck. Here is another similar instance where object
allocation in Resources#subtract will slow down the fair scheduler's event processing thread.
> {noformat}
> org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java)
>     org.apache.hadoop.yarn.util.Records.newRecord(Records.java)
>     org.apache.hadoop.yarn.util.resource.Resources.createResource(Resources.java)
>     org.apache.hadoop.yarn.util.resource.Resources.clone(Resources.java)
>     org.apache.hadoop.yarn.util.resource.Resources.subtract(Resources.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.getResourceUsage(FSAppAttempt.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.getResourceUsage(FSLeafQueue.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
>     java.util.TimSort.binarySort(TimSort.java)
>     java.util.TimSort.sort(TimSort.java)
>     java.util.TimSort.sort(TimSort.java)
>     java.util.Arrays.sort(Arrays.java)
>     java.util.Collections.sort(Collections.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
>     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
>     org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
>     org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
> {noformat}
> One way to fix it is to return {{getCurrentConsumption()}} if there is no preemption
which is the normal case. This means {{getResourceUsage}} method will return reference to
{{FSAppAttempt}}'s internal resource object. But that should be ok as {{getResourceUsage}}
doesn't expect the caller to modify the object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message