hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
Date Thu, 16 Oct 2014 09:32:35 GMT

     [ https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated YARN-2314:
-----------------------------------
    Attachment: tez-yarn-2314.xlsx

Attaching the results of getProxy() call for tez with 20 nodes with this patch for different
cache sizes and for different data sizes (tested a job @200GB and 10 TB scale).  Overall,
there is slight degradation in performance (in milliseconds) by setting cache size to 0, but
not significant to make an impact in overall job runtime in tez.

> ContainerManagementProtocolProxy can create thousands of threads for a large cluster
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-2314
>                 URL: https://issues.apache.org/jira/browse/YARN-2314
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.1.0-beta
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-2314.patch, YARN-2314v2.patch, disable-cm-proxy-cache.patch,
nmproxycachefix.prototype.patch, tez-yarn-2314.xlsx
>
>
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of this cache
is configurable.  However the cache can grow far beyond the configured size when running on
a large cluster and blow AM address/container limits.  More details in the first comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message