Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 93758 invoked from network); 21 Jul 2010 21:08:10 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 Jul 2010 21:08:10 -0000 Received: (qmail 66373 invoked by uid 500); 21 Jul 2010 21:08:08 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 66342 invoked by uid 500); 21 Jul 2010 21:08:07 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 66334 invoked by uid 99); 21 Jul 2010 21:08:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 21:08:07 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender) Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 21:08:01 +0000 DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns; h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic: Thread-Index:Date:Message-ID:References:In-Reply-To: Accept-Language:Content-Language:X-MS-Has-Attach: X-MS-TNEF-Correlator:Content-Type:Content-ID: Content-Transfer-Encoding:MIME-Version; b=llmkZt1tuiGjKB46mxYvzQcGGxGov5vIikfu/pHdDWE913zfZmC5vZmi bYu0vBHiZT5PZuxE05MfvtKH/g83QBZt7Ion9K8k4z0yrm1nDAjqGUyiC RNmGtGar2/5mZIF; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt; s=proddkim; t=1279746481; x=1311282481; h=from:sender:reply-to:subject:date:message-id:to:cc: mime-version:content-transfer-encoding:content-id: content-description:resent-date:resent-from:resent-sender: resent-to:resent-cc:resent-message-id:in-reply-to: references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:list-owner:list-archive; z=From:=20Allen=20Wittenauer=20 |Subject:=20Re:=20How=20to=20control=20the=20number=20of =20map=20tasks=20for=20each=20nodes?|Date:=20Wed,=2021=20 Jul=202010=2021:07:39=20+0000|Message-ID:=20|To:=20""=20 |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted- printable|Content-ID:=20|In-Reply-To:=20|References:=20=0D=0A=20=0D=0A=20=0D=0A=20; bh=qAJR+oK9W08y0wNuzPCaW1yGlBIesuH3xunXtodatQ8=; b=qOIIEqtB9HB5OIl9X0gJHXKa6gHoY8gnfHc0VSoAdcNRPmVeHfGqkaEO hGqxcZ/Y1R4KTAK67RxiA1U4iduu/9HX06z3ZOKp/HqIeylb0fFwaNWnT AjJenYaoD7W9MMu; X-IronPort-AV: E=Sophos;i="4.55,239,1278313200"; d="scan'208";a="13759005" Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Wed, 21 Jul 2010 14:07:39 -0700 From: Allen Wittenauer To: "" Subject: Re: How to control the number of map tasks for each nodes? Thread-Topic: How to control the number of map tasks for each nodes? Thread-Index: AQHLHn0kBiwP9XycEkq/ssP9LmJJ/5KnRUaAgABfqQCAFHPQAIAAUR+A Date: Wed, 21 Jul 2010 21:07:39 +0000 Message-ID: References:

In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org On Jul 21, 2010, at 9:17 AM, Vitaliy Semochkin wrote: > might I ask how did you come to such result? >=20 > In my cluster I use number of mappers and reducers twice more than I have > cpu*cores This is probably a sign that your data is in too many small files. > How did I come to this solution, - first I noticed that in TOP avg load i= s > very law (3-4%) and I noticed that cpu do a lot of WA. >=20 > After several experiments I found out that having number of mappers reduc= ers > TWICE more than I have cpu*core does the best result (the result was almo= st > TWICE BETTER). But that should put more strain on the IO system since now more tasks are w= aiting for input.... so chances are good that your wait isn't IO, but in co= ntext switching.... Another good sign that you have too many files in too = many blocks. > That I can explain by the fast that I do relativly simple log counting > (count number of visitors,hits, etc) > and in this case I have relativly huge amount of IO (logs are huge) and > small amount computation. > I also use mapred.job.reuse.jvm.num.tasks=3D-1 How many files, and what is your block count, and how large is the average = file? 'huge' is fairly relative. :) > What I do not understand is why > mapred.child.java.opts=3D-Xmx256m boosts performance in comparison to -Xm= x160m > how bigger amount of RAM can give me any benefit even if I don't receive = out > of memory errors with smaller -Xmx values?! More memory means that Hadoop doesn't have to spill to disk as often due to= being able to use a larger buffer in RAM.