From user-return-2612-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 1 21:34:53 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ABB14D175 for ; Thu, 1 Nov 2012 21:34:53 +0000 (UTC) Received: (qmail 28750 invoked by uid 500); 1 Nov 2012 21:34:49 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 28634 invoked by uid 500); 1 Nov 2012 21:34:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 11204 invoked by uid 99); 1 Nov 2012 21:30:49 -0000 X-ASF-Spam-Status: No, hits=-2.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_NEUTRAL,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) From: "Cogan, Peter (Peter)" To: Marcos Ortiz CC: "user@hadoop.apache.org" Date: Thu, 1 Nov 2012 22:30:14 +0100 Subject: Re: Set the number of maps Thread-Topic: Set the number of maps Thread-Index: Ac24eBJUJarUsKa6QMC45e24YBUxfg== Message-ID: In-Reply-To: <5092B96B.8020405@uci.cu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CCB89A5F114A5petercoganalcatellucentcom_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.69 on 155.132.188.13 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CCB89A5F114A5petercoganalcatellucentcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Thanks for your answers! From: Marcos Ortiz > Date: Thu, 1 Nov 2012 19:03:23 +0100 To: peter cogan > Cc: "user@hadoop.apache.org" > Subject: Re: Set the number of maps The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximu= m, and like Harsh said to you, is is a TaskTracker service level option. Another thing is that this option is very united to the mapreduce.child.jav= a.opts, so , make sure to monitor constantly the effect of these changes in your cluster. On 11/01/2012 11:55 AM, Harsh J wrote: It can't be set from the code this way - the slot property is applied at the TaskTracker service level (as the name goes). Since you're just testing at the moment, try to set these values, restart TTs, and run your jobs again. You do not need to restart JT at any point for tweaking these values. On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter) wro= te: Hi I understand that the maximum number of concurrent map tasks is set by mapred.tasktracker.map.tasks.maximum - however I wish to run with a smalle= r number of maps (am testing disk IO). I thought that I could set that within= the main program using conf.set("mapred.tasktracker.map.tasks.maximuma", "4"); to run with 4 maps =96 but that seems to have no impact. I know I could jus= t change the mapred-site.xml and restart map reduce but that's kind of a pain= . Can it be set from within the code? Thanks Peter -- Marcos Luis Ort=EDz Valmaseda about.me/marcosortiz @marcosluis2186 [http://universidad.uci.cu/email.gif] --_000_CCB89A5F114A5petercoganalcatellucentcom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Thanks for your = answers!



From: Marcos Ortiz <mlortiz@uci.cu>
Date= : Thu, 1 Nov 2012 19:03:23 +0100
To: peter cogan <peter.cogan@alcatel-lucent.com>
Cc: "user@h= adoop.apache.org" <us= er@hadoop.apache.org>
Subject: <= /span> Re: Set the number of maps

=20 =20 =20
The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.

Another thing is that this option is very united to the mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.



On 11/01/2012 11:55 AM, Harsh J wrote:
It can't be set from the code this way - the slot prop=
erty is applied
at the TaskTracker service level (as the name goes).

Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.

On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Peter.Cogan@alcatel-lucent.com> wrote:
Hi

I understand that the maximum number of concurrent map tasks is set by
mapred.tasktracker.map.tasks.maximum  - however I wish to run with a smalle=
r
number of maps (am testing disk IO). I thought that I could set that within=
the main program using

conf.set("mapred.tasktracker.map.tasks.maximuma", "4");


to run with 4 maps =96 but that seems to have no impact. I know I could jus=
t
change the mapred-site.xml and restart map reduce but that's kind of a pain=
.
Can it be set from within the code?


Thanks

Peter

    

--
=20
=20

=
--_000_CCB89A5F114A5petercoganalcatellucentcom_--