Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9294417AD2 for ; Mon, 13 Oct 2014 22:05:37 +0000 (UTC) Received: (qmail 32414 invoked by uid 500); 13 Oct 2014 22:05:37 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 32358 invoked by uid 500); 13 Oct 2014 22:05:37 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 32346 invoked by uid 99); 13 Oct 2014 22:05:37 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2014 22:05:37 +0000 Date: Mon, 13 Oct 2014 22:05:37 +0000 (UTC) From: "Jason Lowe (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170082#comment-14170082 ] Jason Lowe commented on YARN-2314: ---------------------------------- The patch effectively restores 0.23 behavior in this area, as 0.23 also had the client close the connection after each RPC call and did not attempt to cache proxies. We were happy with the performance from 0.23, and we did not measure any significant slowdown in this area for 2.x once this patch was applied. > ContainerManagementProtocolProxy can create thousands of threads for a large cluster > ------------------------------------------------------------------------------------ > > Key: YARN-2314 > URL: https://issues.apache.org/jira/browse/YARN-2314 > Project: Hadoop YARN > Issue Type: Bug > Components: client > Affects Versions: 2.1.0-beta > Reporter: Jason Lowe > Priority: Critical > Attachments: disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch > > > ContainerManagementProtocolProxy has a cache of NM proxies, and the size of this cache is configurable. However the cache can grow far beyond the configured size when running on a large cluster and blow AM address/container limits. More details in the first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)