Return-Path: X-Original-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5D58C17BDC for ; Wed, 4 Feb 2015 03:52:19 +0000 (UTC) Received: (qmail 36588 invoked by uid 500); 4 Feb 2015 03:52:20 -0000 Delivered-To: apmail-hadoop-yarn-dev-archive@hadoop.apache.org Received: (qmail 36526 invoked by uid 500); 4 Feb 2015 03:52:19 -0000 Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-dev@hadoop.apache.org Delivered-To: mailing list yarn-dev@hadoop.apache.org Received: (qmail 36514 invoked by uid 99); 4 Feb 2015 03:52:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 03:52:19 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lavkesh51@gmail.com designates 209.85.220.54 as permitted sender) Received: from [209.85.220.54] (HELO mail-pa0-f54.google.com) (209.85.220.54) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 03:52:13 +0000 Received: by mail-pa0-f54.google.com with SMTP id eu11so104885948pac.13 for ; Tue, 03 Feb 2015 19:51:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=e3I66+QumcOmKczwRMeD00Z7pjaHCejvSSc1q66nfzo=; b=LXSO0LA6XYJ4Hhj8WXCf6LlS6zmWgWzw29/y3EBPEVTcsGo+0du5ACHZQslT4QAFvf 0hCrfb/wnckZ4pEAEzEG+sftIdGQWXeFu0DTrTFy7BwWKJ8OE9Xz6FfnRhdNW1R7jgZy RK7tPgDEDs+Yw/DmUNZpb+0zlghBuBelCPyXNOpzl6TmR41YCkM14uYrYiLQVGsw647O JXeTChKCG99bSI2y9JGjymj4MAdOZKpA1IC64kTKKq32nCLEcOaVPUMYX20ejZDImOY4 uStwg90mSytxkwrVYJXlf9R5R/V9iyrUkZkRBIrET81tBUZlSUnrFXlJcSXrMRtf5uVY 7eyw== MIME-Version: 1.0 X-Received: by 10.67.1.132 with SMTP id bg4mr41991636pad.151.1423021913498; Tue, 03 Feb 2015 19:51:53 -0800 (PST) Sender: lavkesh51@gmail.com Received: by 10.70.31.3 with HTTP; Tue, 3 Feb 2015 19:51:53 -0800 (PST) Date: Wed, 4 Feb 2015 09:21:53 +0530 X-Google-Sender-Auth: l1pU4dHGh2iWeyjmokNkG3akY3s Message-ID: Subject: Tuning thread counts in yarn production cluster. From: Lavkesh Lahngir To: yarn-dev@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b5dbd28a96bca050e3b1b84 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5dbd28a96bca050e3b1b84 Content-Type: text/plain; charset=UTF-8 Needed suggestions on how to arrive at thread counts for Yarn cluster. Hadoop version: 2.6.0 Number of nodes in cluster : 250 . Node configuration: 24cores, 64 GB RAM and 6 * 1 TB (7200 RPM) Expected number of jobs at peak time: 1500 Maximum number of containers required by a job: 80k-100k We are trying to tune various thread counts. yarn.resourcemanager.client.thread-count yarn.resourcemanager.scheduler.client.thread-count yarn.resourcemanager.resource-tracker.client.thread-count yarn.nodemanager.container-manager.thread-count yarn.app.mapreduce.am.job.task.listener.thread-count Thanks --047d7b5dbd28a96bca050e3b1b84--