Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of anilgupta84@gmail.com
 designates 209.85.215.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOcnVr3mbqGHOpiptB=eazqM0K0uePmvcJWs-xnHowDtT5ik_g@mail.gmail.com>
References: 
 <CAF1+Vs8avj3vTgerG=20O5Ai_aHk+rVk+9vx_EVJX6sZkNnvjA@mail.gmail.com>
 <CAOcnVr3mbqGHOpiptB=eazqM0K0uePmvcJWs-xnHowDtT5ik_g@mail.gmail.com>
From: anil gupta <anilgupta84@gmail.com>
Date: Wed, 26 Dec 2012 10:41:43 -0800
Message-ID: 
 <CAF1+Vs9dtgn_qMzbzLA-txUNhrmZrWMPp41hLfThDzdGtzViFQ@mail.gmail.com>
Subject: Re: Setting number of mappers in Teragen
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b621df85e638c04d1c5c99d

--047d7b621df85e638c04d1c5c99d
Content-Type: text/plain; charset=ISO-8859-1

Hi Harsh,

Fixed it. I was putting the -Dmapred.map.tasks=20 after specifying the
input directory. I completely forgot about this trick of
genericOptionParser of Hadoop. Thanks a lot. :)

On Wed, Dec 26, 2012 at 10:33 AM, Harsh J <harsh@cloudera.com> wrote:

> The MR1 teragen's mappers # depends on the total number of rows and
> demanded # of maps.
>
> How are you passing -Dmapred.map.tasks=20 (no spaces) exactly? All
> generic options must go in before any other options do, so it should
> appear right after the word "teragen" in your command.
>
> On Wed, Dec 26, 2012 at 11:49 PM, anil gupta <anilgupta84@gmail.com>
> wrote:
> > Hi All,
> >
> > I have 5 worker nodes and i have 4 map slots per node. So, i have 20 map
> > slots in my cluster. But when, i start my Teragen job, it only spawns 2
> > mappers for entire job. I have even tried using the option
> > -Dmapred.map.tasks = 20 . Can anyone tell me how to force teragen to use
> 20
> > mappers for generating the data? I am using cdh4.1.2 with
> Mapreducev1(Hadoop
> > 0.20.2)
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
>
> --
> Harsh J
>


-- 
Thanks & Regards,
Anil Gupta

--047d7b621df85e638c04d1c5c99d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Harsh,<br><br>Fixed it. I was putting the -Dmapred.map.tasks=3D20 after =
specifying the input directory. I completely forgot about this trick of gen=
ericOptionParser of Hadoop. Thanks a lot. :)<br><br><div class=3D"gmail_quo=
te">

On Wed, Dec 26, 2012 at 10:33 AM, Harsh J <span dir=3D"ltr">&lt;<a href=3D"=
mailto:harsh@cloudera.com" target=3D"_blank">harsh@cloudera.com</a>&gt;</sp=
an> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex">

The MR1 teragen&#39;s mappers # depends on the total number of rows and<br>
demanded # of maps.<br>
<br>
How are you passing -Dmapred.map.tasks=3D20 (no spaces) exactly? All<br>
generic options must go in before any other options do, so it should<br>
appear right after the word &quot;teragen&quot; in your command.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
On Wed, Dec 26, 2012 at 11:49 PM, anil gupta &lt;<a href=3D"mailto:anilgupt=
a84@gmail.com">anilgupta84@gmail.com</a>&gt; wrote:<br>
&gt; Hi All,<br>
&gt;<br>
&gt; I have 5 worker nodes and i have 4 map slots per node. So, i have 20 m=
ap<br>
&gt; slots in my cluster. But when, i start my Teragen job, it only spawns =
2<br>
&gt; mappers for entire job. I have even tried using the option<br>
&gt; -Dmapred.map.tasks =3D 20 . Can anyone tell me how to force teragen to=
 use 20<br>
&gt; mappers for generating the data? I am using cdh4.1.2 with Mapreducev1(=
Hadoop<br>
&gt; 0.20.2)<br>
&gt; --<br>
&gt; Thanks &amp; Regards,<br>
&gt; Anil Gupta<br>
<br>
<br>
<br>
</div></div><span class=3D"HOEnZb"><font color=3D"#888888">--<br>
Harsh J<br>
</font></span></blockquote></div><br><br clear=3D"all"><br>-- <br>Thanks &a=
mp; Regards,<br>Anil Gupta

--047d7b621df85e638c04d1c5c99d--