hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6679) Reduce Resource instance overhead via non-PBImpl
Date Thu, 08 Jun 2017 19:54:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043321#comment-16043321

Wangda Tan commented on YARN-6679:


I just tried to test the patch using SLS: (a generated trace which consists of 8K nodes, each
node has 128GB memory, 400 concurrent running apps, size of each app varies).

>From my test, YARN scheduler can allocate 2700-3100 containers / sec on latest trunk with
and without the patch, I didn't see obvious regression however I didn't see obvious improvement
either. Not sure if it is the same for branch-2.8 as we have already fixed some performance
issues on trunk.

I don't have chance to look into the issue since theoretically this patch should improve performance.
Not sure if we hit any other bottleneck, [~daryn], could you share size / allocation-throughput
of your cluster?

> Reduce Resource instance overhead via non-PBImpl
> ------------------------------------------------
>                 Key: YARN-6679
>                 URL: https://issues.apache.org/jira/browse/YARN-6679
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.8.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 2.9.0, 3.0.0-alpha4
>         Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, YARN-6679.3.branch-2.patch,
YARN-6679.3.trunk.patch, YARN-6679.branch-2.patch, YARN-6679.trunk.patch
> Creating and using transient PB-based Resource instances during scheduling is very expensive.
 The overhead can be transparently reduced by internally using lightweight non-PB based instances.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message