From user-return-1771-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 2 17:39:35 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 18A83D419 for ; Tue, 2 Oct 2012 17:39:35 +0000 (UTC) Received: (qmail 76942 invoked by uid 500); 2 Oct 2012 17:39:30 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 76768 invoked by uid 500); 2 Oct 2012 17:39:30 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 76760 invoked by uid 99); 2 Oct 2012 17:39:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Oct 2012 17:39:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.138.91.141] (HELO nm11-vm3.bullet.mail.ne1.yahoo.com) (98.138.91.141) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Oct 2012 17:39:21 +0000 Received: from [98.138.90.56] by nm11.bullet.mail.ne1.yahoo.com with NNFMP; 02 Oct 2012 17:38:59 -0000 Received: from [98.138.86.157] by tm9.bullet.mail.ne1.yahoo.com with NNFMP; 02 Oct 2012 17:38:59 -0000 Received: from [127.0.0.1] by omp1015.mail.ne1.yahoo.com with NNFMP; 02 Oct 2012 17:38:59 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 952703.9616.bm@omp1015.mail.ne1.yahoo.com Received: (qmail 82891 invoked by uid 60001); 2 Oct 2012 17:38:59 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1349199539; bh=9dqui2Xpvtnxa+/XbtsVU56cDREw3lZmdOjTknpuEbs=; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=Dr1BP7L1SfNp9Lg7DqylVYx9WBJXF3fOFdyD4unev3N8crmv7aRUzUQIbmGelkMiPRL9qWBHrlJ4dwAdP8Gd+zWsXNsbecjLuMFy6fEOHP+y2e4aiWybINlhmd3O6lgcYOUFrHXQtSJl/+jaLX4/X0RMxGtEkCqOqWN4TUEWQK4= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=HHA0SAbHHVWcYLHUQO65Znr5BM21ZSrF35ELVDgmVsVq4+kT6hvMDeD6ZoY53W8L6+c9JIQyFXQVEeXKn9veGZ8Pgm9FNrpR9buoGfhacK353zYWoZmO13ExLbc5RR5acdPwugVFkX/Q1GVszBrxwgzdWjeRKFWUq9ZYj/L+xaQ=; X-YMail-OSG: Aiuo.7cVM1mDPhwYJX38Nd1rAwp_e0phE.DO2JyywF86DZJ oTKCZgwvCSofNFEoNlSMt1tJyYsAyZu70xXPo6CIYcVGhlMbTskIOjpnnXaa XaqbS6o.N079UAV0EQWTsqi9rSMXy9hDSkpZm4DpCv6_USAdzAMYt0xProBc .qWlDzgdafnsqV.6fL6.K7shI8qwkkdwOvRb.A.RliCAQI7nHfWW4.WKK_4V wLKPDBeHNGmaP6HqkcAFx5zsC6Y9WTL9pFiFtprBteCUKfOVCbkkXu7lWyel k.bkN3Ve7LDpJncQjP9D_fBeU3V4Ym9Adoiglto5.lxdUhF_UrQ4n7RuecSn b_fh3GLwbxiaCjxgATWgpZxM39tctz9q4yo79QeZCBfUjUm0LJ_d_K3486RL re.EG9kC50siJbsIfXu1xbkhiV_f9koUfwHYvS52UtzlTHnVxBxaqHQNk1Ie 8.Ymd8eRrSXE4bLiAR2JS0QOxd1.wpeEscOmhrvOR7EgAbsE44pEbqLKkA.c iiEvX Received: from [62.49.31.174] by web125304.mail.ne1.yahoo.com via HTTP; Tue, 02 Oct 2012 10:38:59 PDT X-Mailer: YahooMailWebService/0.8.121.434 References: <1349195676.38555.YahooMailNeo@web125301.mail.ne1.yahoo.com> Message-ID: <1349199539.93871.YahooMailNeo@web125304.mail.ne1.yahoo.com> Date: Tue, 2 Oct 2012 10:38:59 -0700 (PDT) From: Shing Hing Man Reply-To: Shing Hing Man Subject: Re: How to lower the total number of map tasks To: "user@hadoop.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-1622010646-1284396435-1349199539=:93871" X-Virus-Checked: Checked by ClamAV on apache.org ---1622010646-1284396435-1349199539=:93871 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable =0AI have tried =0A=A0=A0=A0=A0=A0=A0 Configuration.setInt("mapred.max.spli= t.size",134217728);=0A=0Aand setting mapred.max.split.size in mapred-site.x= ml. ( dfs.block.size is left unchanged at 67108864).=0A=0ABut in the job.xm= l, I am still getting mapred.map.tasks =3D242 .=0A=0AShing =0A=0A=0A=0A=0A= =0A=0A________________________________=0A From: Bejoy Ks =0ATo: user@hadoop.apache.org; Shing Hing Man =0AS= ent: Tuesday, October 2, 2012 6:03 PM=0ASubject: Re: How to lower the total= number of map tasks=0A =0A=0ASorry for the typo, the property name is=A0ma= pred.max.split.size=0A=0AAlso just for=A0changing=A0the number of map tasks= you don't need to modify the hdfs block size.=0A=0A=0AOn Tue, Oct 2, 2012 = at 10:31 PM, Bejoy Ks wrote:=0A=0AHi=0A>=0A>=0A>Yo= u need to alter the value of mapred.max.split size to a value larger than y= our block size to have less number of map tasks than the default.=0A>=0A>= =0A>=0A>On Tue, Oct 2, 2012 at 10:04 PM, Shing Hing Man = wrote:=0A>=0A>=0A>>=0A>>=0A>>I am running Hadoop 1.0.3 in Pseudo=A0=A0distr= ibuted mode.=0A>>When I=A0=A0submit a map/reduce job to process a file of= =A0=A0size about 16 GB, in job.xml, I have the following=0A>>=0A>>=0A>>mapr= ed.map.tasks =3D242=0A>>mapred.min.split.size =3D0=0A>>dfs.block.size =3D 6= 7108864=0A>>=0A>>=0A>>I would like to reduce=A0=A0 mapred.map.tasks to see = if it improves performance.=0A>>I have tried doubling=A0=A0the size of=A0= =A0dfs.block.size. But the=A0=A0=A0=A0mapred.map.tasks remains unchanged.= =0A>>Is there a way to reduce=A0=A0mapred.map.tasks=A0=A0?=0A>>=0A>>=0A>>Th= anks in advance for any assistance ! =A0=0A>>Shing=0A>>=0A>>=0A> ---1622010646-1284396435-1349199539=:93871 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable

I have tried
=        Configuration.setInt("mapred.max.split= .size",134217728);

and setting mapred.max.split.size in mapred-site.= xml. ( dfs.block.size is left unchanged at 67108864).

But in the job= .xml, I am still getting mapred.map.tasks =3D242 .

Shing




From: Bejoy Ks <bejoy.hadoop@gmail.com>
To: user@hadoop.apache.org; Shing Hing Man <mat= msh@yahoo.com>
Sent: Tuesday, October 2, 2012 6:03 PM
Subject:= Re: How to lower the total number of map tasks

Sorry for the typo, the property name is&n= bsp;mapred.max.split.size

Also just for changing&nb= sp;the number of map tasks you don't need to modify the hdfs block size.
On Tue, Oct 2, 2012 at 10:31 P= M, Bejoy Ks <bejoy.hadoop@gmail.com> wrote:
=0A
Hi

You need to alter the value o= f mapred.max.split size to a value larger than your block size to have less= number of map tasks than the default.
=0A


On Tue, Oct 2, 2012 at 10= :04 PM, Shing Hing Man <matmsh@yahoo.com> wrote:
=0A

=0A
=0A
=0AI am running Hadoop 1.0.3 in Pseudo&n= bsp; distributed mode.
=0AWhen I  submit a map/reduce job= to process a file of  size about 16 GB, in job.xml, I have the f= ollowing
=0A
=0A
=0Amapred.map.tasks =3D242
=0Amapred.min.split= .size =3D0
=0Adfs.block.size =3D 67108864
=0A
=0A
=0AI would li= ke to reduce   mapred.map.tasks to see if it improves performance= .
=0AI have tried doubling  the size of  dfs.block.s= ize. But the    mapred.map.tasks remains unchanged.
= =0AIs there a way to reduce  mapred.map.tasks  ?
=0A=
=0A
=0AThanks in advance for any assistance !  
=0AShing
=0A
=0A
=0A

=0A


---1622010646-1284396435-1349199539=:93871--