Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EBB6610869 for ; Tue, 30 Apr 2013 10:52:31 +0000 (UTC) Received: (qmail 34042 invoked by uid 500); 30 Apr 2013 10:52:27 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 33584 invoked by uid 500); 30 Apr 2013 10:52:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 33562 invoked by uid 99); 30 Apr 2013 10:52:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Apr 2013 10:52:25 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com designates 209.85.215.46 as permitted sender) Received: from [209.85.215.46] (HELO mail-la0-f46.google.com) (209.85.215.46) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Apr 2013 10:52:18 +0000 Received: by mail-la0-f46.google.com with SMTP id fs13so299801lab.33 for ; Tue, 30 Apr 2013 03:51:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=KgrLwQE4NyGGoQ7pz7j4XbcECHt96zeX/+o3cCpZAPM=; b=I3meYJgxmQDVJ5EdmOp5cCbhaZ6i2G61DXmXlRiUoUsXlDddMo+sj4e2GfA6WGz8k1 dxccoeJLQ67d+okhQHzWDqWBrw4x7iyGIzNiVbz7KDcW74QX1cHC1DlPeeR9PFNgJmO6 YeDIXmnNjOtCPhu6NZsJeUgWjI0+vRG7oPC7vVJWdVgva1Y8WWjc3mIrtSrKPNHJzpAi DFWZZ30BeCzH2Kkpm83H7UkdvNNW38BmWB7e0R5hjs/tclM5phHzri1U/l5hVF7SUMSF r7LlYcLsfCqrynoPzB/BfASGpm1BVtnTIlNbhEbEyNyOMwvuIPBUjW9O+RO66sfyHFt+ xpOQ== MIME-Version: 1.0 X-Received: by 10.112.160.226 with SMTP id xn2mr2129829lbb.16.1367318740531; Tue, 30 Apr 2013 03:45:40 -0700 (PDT) Received: by 10.114.24.129 with HTTP; Tue, 30 Apr 2013 03:45:40 -0700 (PDT) In-Reply-To: References: Date: Tue, 30 Apr 2013 16:15:40 +0530 Message-ID: Subject: Re: Set reducer capacity for a specific M/R job From: Nitin Pawar To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c37c44d3020604db91b30b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c37c44d3020604db91b30b Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I don't think you can control how many reducers can run parallely via framework. Other way to do this is increase the memory given to individual reducer so that the tasktracker will be limited by memory to launch more reducers at the same time and they will queue up you can try setting up this mapred.job.reduce.memory.mb to a higher value and see if that works On Tue, Apr 30, 2013 at 4:08 PM, Han JU wrote: > Yes.. In the conf file of my cluster, mapred.tasktracker.reduce.tasks.max= imum > is 8. > And for this job, I want it to be 4. > I set it through conf and build the job with this conf, then submit it. > But hadoop lauches 8 reduce per datanode... > > > 2013/4/30 Nitin Pawar > >> so basically if I understand correctly >> >> you want to limit the # parallel execution of reducers only for this job= ? >> >> >> >> On Tue, Apr 30, 2013 at 4:02 PM, Han JU wrote: >> >>> Thanks. >>> >>> In fact I don't want to set reducer or mapper numbers, they are fine. >>> I want to set the reduce slot capacity of my cluster when it executes m= y >>> specific job. Say I have 100 reduce tasks for this job, I want my clust= er >>> to execute 4 of them in the same time, not 8 of them in the same time, = only >>> for this specific job. >>> So I set mapred.tasktracker.reduce.tasks.maximum to 4 and submit the >>> job. This conf is well received by the job, but ignored by hadoop .. >>> >>> Any idea why is this? >>> >>> >>> 2013/4/30 Nitin Pawar >>> >>>> The *mapred*.*tasktracker*.*reduce*.*tasks*.*maximum* parameter sets >>>> the maximum number of reduce tasks that may be run by an individual >>>> TaskTracker server at one time. This is not per job configuration. >>>> >>>> he number of map tasks for a given job is driven by the number of inpu= t >>>> splits and not by the mapred.map.tasks parameter. For each input split= a >>>> map task is spawned. So, over the lifetime of a mapreduce job the numb= er of >>>> map tasks is equal to the number of input splits. mapred.map.tasks is = just >>>> a hint to the InputFormat for the number of maps >>>> >>>> If you want to set max number of maps or reducers per job then you can >>>> set the hints by using the job object you created >>>> job.setNumMapTasks() >>>> >>>> Note this is just a hint and again the number will be decided by the >>>> input split size. >>>> >>>> >>>> On Tue, Apr 30, 2013 at 3:39 PM, Han JU wrote= : >>>> >>>>> Thanks Nitin. >>>>> >>>>> What I need is to set slot only for a specific job, not for the whole >>>>> cluster conf. >>>>> But what I did does NOT work ... Have I done something wrong? >>>>> >>>>> >>>>> 2013/4/30 Nitin Pawar >>>>> >>>>>> The config you are setting is for job only >>>>>> >>>>>> But if you want to reduce the slota on tasktrackers then you will >>>>>> need to edit tasktracker conf and restart tasktracker >>>>>> On Apr 30, 2013 3:30 PM, "Han JU" wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I want to change the cluster's capacity of reduce slots on a per jo= b >>>>>>> basis. Originally I have 8 reduce slots for a tasktracker. >>>>>>> I did: >>>>>>> >>>>>>> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4"); >>>>>>> ... >>>>>>> Job job =3D new Job(conf, ...) >>>>>>> >>>>>>> >>>>>>> And in the web UI I can see that for this job, the max reduce tasks >>>>>>> is exactly at 4, like I set. However hadoop still launches 8 reduce= r per >>>>>>> datanode ... why is this? >>>>>>> >>>>>>> How could I achieve this? >>>>>>> -- >>>>>>> *JU Han* >>>>>>> >>>>>>> Software Engineer Intern @ KXEN Inc. >>>>>>> UTC - Universit=E9 de Technologie de Compi=E8gne >>>>>>> * **GI06 - Fouille de Donn=E9es et D=E9cisionnel* >>>>>>> >>>>>>> +33 0619608888 >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *JU Han* >>>>> >>>>> Software Engineer Intern @ KXEN Inc. >>>>> UTC - Universit=E9 de Technologie de Compi=E8gne >>>>> * **GI06 - Fouille de Donn=E9es et D=E9cisionnel* >>>>> >>>>> +33 0619608888 >>>>> >>>> >>>> >>>> >>>> -- >>>> Nitin Pawar >>>> >>> >>> >>> >>> -- >>> *JU Han* >>> >>> Software Engineer Intern @ KXEN Inc. >>> UTC - Universit=E9 de Technologie de Compi=E8gne >>> * **GI06 - Fouille de Donn=E9es et D=E9cisionnel* >>> >>> +33 0619608888 >>> >> >> >> >> -- >> Nitin Pawar >> > > > > -- > *JU Han* > > Software Engineer Intern @ KXEN Inc. > UTC - Universit=E9 de Technologie de Compi=E8gne > * **GI06 - Fouille de Donn=E9es et D=E9cisionnel* > > +33 0619608888 > --=20 Nitin Pawar --001a11c37c44d3020604db91b30b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I don't think you can control = how many reducers can run parallely via framework.=A0

Other way= to do this is increase the memory given to individual reducer so that the = tasktracker will be limited by memory to launch more reducers at the same t= ime and they will queue up=A0

you can try setting up this=A0mapred.job.reduce.memory.mb to a higher value and see if that works


On Tue, Apr 3= 0, 2013 at 4:08 PM, Han JU <ju.han.felix@gmail.com> wro= te:
Yes.. In the conf file of m= y cluster,=A0ma= pred.tasktracker.reduce.tasks.maximum is 8.
And for this jo= b, I want it to be 4.
I set it=A0through=A0conf and build the job wit= h this conf, then submit it. But hadoop lauches 8 reduce per datanode...

2013/4/30 Nitin Pawar &l= t;nitinpawar43= 2@gmail.com>
so basically if I understand correctly=A0

you want to limit the # parallel execution of reducers only for this job= ?=A0



On Tue, Apr 30, 2013 at 4:02 PM, Han JU <ju.han.felix@gmail.com&g= t; wrote:
Thanks.

In fact I don't want = to set reducer or mapper numbers, they are fine.
I want to set the redu= ce slot capacity of my cluster when it executes my specific job. Say I have= 100 reduce tasks for this job, I want my cluster to execute 4 of them in t= he same time, not 8 of them in the same time, only for this specific job.
So I set=A0mapred.tasktracker.reduce.tasks.maximum to 4 and submit the= job. This conf is well received by the job, but ignored by hadoop ..
=

Any idea why is this?


2013/4/30 Nitin Pawar = <nitinpawar= 432@gmail.com>
The=A0mapred.tasktracker.reduce.<= em style=3D"font-weight:bold;font-style:normal;color:rgb(68,68,68);line-hei= ght:16px">tasks.<= /span>maximum=A0parameter sets the maximum number of reduce tasks that may be run= by an individual TaskTracker server at one time. This is not per job confi= guration.=A0

he number of map tasks for a= given job is driven by the number of input splits and not by the mapred.ma= p.tasks parameter. For each input split a map task is spawned. So, over the= lifetime of a mapreduce job the number of map tasks is equal to the number= of input splits. mapred.map.tasks is just a hint to the InputFormat for th= e number of maps

If you want to set max= number of maps or reducers per job then you can set the hints by using the= job object you created=A0
job.setNumMapTasks()

Note= this is just a hint and again the number will be decided by the input spli= t size.=A0


On Tue, Apr 30, 2013 at 3:39 PM, Han JU <ju.han.felix@gmail.com&g= t; wrote:
Thanks Nitin.

What I need is to set slo= t only for a specific job, not for the whole cluster conf.
But wh= at I did does NOT work ... Have I done something wrong?


2013/4/30 Nit= in Pawar <nitinpawar432@gmail.com>

The config you are setting is for job only

But if you want to reduce the slota on tasktrackers then you= will need to edit tasktracker conf and restart tasktracker

On Apr 30, 2013 3:30 PM, "Han JU" <= ju.han.felix@gm= ail.com> wrote:
Hi,

I want to change the cluster's = capacity of reduce slots on a per job basis. Originally I have 8 reduce slo= ts for a tasktracker.
I did:

conf.set("mapred.tasktracker.reduce.tasks.maximum", "4"= );
...
Job job =3D new Job(conf, ...)

And in the web UI I can see that for this job, the max reduce tasks is exa= ctly at 4, like I set. However hadoop still=A0launches 8 reducer per datano= de ... why is this?

How could I=A0achieve=A0this?
--
JU Han
Sof= tware Engineer Intern @ KXEN Inc.
UTC=A0=A0 - =A0Universit=E9 de Technologie de Compi=E8gne
=A0=A0=A0=A0 GI06 - Fouille de Donn=E9es et D=E9ci= sionnel




--
=
JU Han
UTC=A0=A0 - =A0Universit=E9 de Technologie de Compi=E8gne
=A0=A0=A0=A0 GI06 - Fouille de Donn=E9es et D=E9ci= sionnel




<= /div>--
Nitin Pawar



--
JU Han

=
Software E= ngineer Intern @ KXEN Inc.
UTC=A0=A0 - =A0Universit=E9 de Technologie de Compi=E8gne
=A0=A0=A0=A0 GI06 - Fouille de Donn=E9es et D=E9ci= sionnel




<= /div>--
Nitin Pawar


--
JU Han

=
Software E= ngineer Intern @ KXEN Inc.
UTC=A0=A0 - =A0Universit=E9 de Technologie de Compi=E8gne
=A0=A0=A0=A0 GI06 - Fouille de Donn=E9es et D=E9ci= sionnel




--
= Nitin Pawar
--001a11c37c44d3020604db91b30b--