Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0377411548 for ; Wed, 27 Aug 2014 11:16:07 +0000 (UTC) Received: (qmail 71004 invoked by uid 500); 27 Aug 2014 11:15:33 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 70875 invoked by uid 500); 27 Aug 2014 11:15:33 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70865 invoked by uid 99); 27 Aug 2014 11:15:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 11:15:33 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [157.193.49.127] (HELO smtp3.ugent.be) (157.193.49.127) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 11:15:05 +0000 Received: from localhost (mcheck2.ugent.be [157.193.49.249]) by smtp3.ugent.be (Postfix) with ESMTP id 490C0C3A3 for ; Wed, 27 Aug 2014 13:15:04 +0200 (CEST) X-Virus-Scanned: by UGent DICT Received: from smtp3.ugent.be ([IPv6:::ffff:157.193.49.127]) by localhost (mcheck2.UGent.be [::ffff:157.193.43.11]) (amavisd-new, port 10024) with ESMTP id K3dEOP1boqjy for ; Wed, 27 Aug 2014 13:15:04 +0200 (CEST) Received: from localhost.localdomain (spike.ugent.be [157.193.44.183]) (Authenticated sender: stdweird) by smtp3.ugent.be (Postfix) with ESMTPSA id 9607CC3A1 for ; Wed, 27 Aug 2014 13:15:03 +0200 (CEST) Message-ID: <53FDBDA3.2080700@ugent.be> Date: Wed, 27 Aug 2014 13:14:43 +0200 From: Stijn De Weirdt User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: total number of map tasks Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Miltered: at jchkm3 with ID 53FDBDA3.001 by Joe's j-chkmail (http://helpdesk.ugent.be/email/)! X-j-chkmail-Enveloppe: 53FDBDA3.001 from spike.ugent.be/spike.ugent.be/157.193.44.183/localhost.localdomain/ X-j-chkmail-Score: MSGID : 53FDBDA3.001 on smtp3.ugent.be : j-chkmail score : . : R=. U=. O=. B=0.000 -> S=0.000 X-j-chkmail-Status: Ham X-Virus-Checked: Checked by ClamAV on apache.org hi all, we are tuning yarn (or trying to) on our environment (shared fielsystem, no hdfs) using terasort and one of the main issue we are seeing is that an avg map task takes < 15sec. some tuning guides and websites suggest that ideally map tasks run between 40sec to 1 or 2 minutes. (however, it's also not very clear if the recommendations are still valid for yarn) in particluar, we see way more map tasks then expected, and we are wondering how the number of map tasks per job run is determined. teragen created 64 output files, we are only expecting 64 map tasks, each processing one input file. however, we see something like 3000 tasks hints are much appreciated stijn