hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
Date Thu, 01 Jun 2017 20:48:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033655#comment-16033655
] 

Daryn Sharp commented on YARN-6679:
-----------------------------------

Thanks Daniel!

Using signum to handle a NaN (impossible, I hope?) may not be worth the cost.   I only moved
the pre-existing method from the pb impl to the base class so I'd suggest another jira if
feel strongly that it should be changed?

bq. How thoroughly has this been tested?
I'd say very.  The first large/busy pre-production cluster was crippled by 2.8.  The scheduler
thread was constantly pegging a cpu and falling behind.  Deploys were halted.  We deployed
my collection of patches about 1.5w ago.  Cpu fluctuates a lot, but doesn't stay pegged anymore.

bq. Wanna add some unit tests to confirm the newInstance() methods and the PB conversion work
as expected?
I would certainly hope the existing tests provide coverage! :)  I didn't expose any new methods
to test but I'll concoct some rudimentary tests if need be.

> Reduce Resource instance overhead via non-PBImpl
> ------------------------------------------------
>
>                 Key: YARN-6679
>                 URL: https://issues.apache.org/jira/browse/YARN-6679
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.8.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: YARN-6679.branch-2.patch, YARN-6679.trunk.patch
>
>
> Creating and using transient PB-based Resource instances during scheduling is very expensive.
 The overhead can be transparently reduced by internally using lightweight non-PB based instances.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message