Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of abhay.ratnaparkhi@gmail.com
 designates 74.125.83.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACD21EODWvkJ8z4ytGG4iRQo7KZjW2i=vFAJS5OnE6MfZ1E+Cw@mail.gmail.com>
References: 
 <CAM2hXiUAJfPiR6423kKioSo1=xm0iBwpEU+6QvXC_qPxcXOiXA@mail.gmail.com>
	<CACD21EODWvkJ8z4ytGG4iRQo7KZjW2i=vFAJS5OnE6MfZ1E+Cw@mail.gmail.com>
Date: Mon, 3 Sep 2012 21:06:47 +0530
Message-ID: 
 <CAM2hXiU5VTGZtsfahLOmMT3bZu6MUKsXotYLgJqvwS8fnJPDpg@mail.gmail.com>
Subject: Re: knowing the nodes on which reduce tasks will run
From: Abhay Ratnaparkhi <abhay.ratnaparkhi@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b343958d7df4704c8cde87f

--047d7b343958d7df4704c8cde87f
Content-Type: text/plain; charset=ISO-8859-1

How can I set  'mapred.tasktracker.reduce.tasks.maximum'  to "0" in a
running tasktracker?
Seems that I need to restart the tasktracker and in that case I'll loose
the output of map tasks by particular tasktracker.

Can I change   'mapred.tasktracker.reduce.tasks.maximum'  to "0"  without
restarting tasktracker?

~Abhay

On Mon, Sep 3, 2012 at 8:53 PM, Bejoy Ks <bejoy.hadoop@gmail.com> wrote:

> HI Abhay
>
> The TaskTrackers on which the reduce tasks are triggered is chosen in
> random based on the reduce slot availability. So if you don't need the
> reduce tasks to be scheduled on some particular nodes you need to set
> 'mapred.tasktracker.reduce.tasks.maximum' on those nodes to 0. The
> bottleneck here is that this property is not a job level one you need to
> set it on a cluster level.
>
> A cleaner approach will be to configure each of your nodes with the right
> number of map and reduce slots based on the resources available on each
> machine.
>
>
> On Mon, Sep 3, 2012 at 7:49 PM, Abhay Ratnaparkhi <
> abhay.ratnaparkhi@gmail.com> wrote:
>
>> Hello,
>>
>> How can one get to know the nodes on which reduce tasks will run?
>>
>> One of my job is running and it's completing all the map tasks.
>> My map tasks write lots of intermediate data. The intermediate directory
>> is getting full on all the nodes.
>> If the reduce task take any node from cluster then It'll try to copy the
>> data to same disk and it'll eventually fail due to Disk space related
>> exceptions.
>>
>> I have added few more tasktracker nodes in the cluster and now want to
>> run reducer on new nodes only.
>> Is it possible to choose a node on which the reducer will run? What's the
>> algorithm hadoop uses to get a new node to run reducer?
>>
>> Thanks in advance.
>>
>> Bye
>> Abhay
>>
>
>

--047d7b343958d7df4704c8cde87f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

How can I set=A0
&#39;mapred.tasktracker.reduce.tasks.maximum&#39;=A0 to &quot;0&quot; in a =
running tasktracker?<div>Seems that I need to restart the tasktracker and i=
n that case I&#39;ll loose the output of map tasks by particular tasktracke=
r.</div>
<div><br></div><div>Can I change=A0
=A0&#39;mapred.tasktracker.reduce.tasks.maximum&#39;=A0 to &quot;0&quot;=A0=
 without restarting tasktracker?</div><div><br></div><div>~Abhay<br><br><di=
v class=3D"gmail_quote">On Mon, Sep 3, 2012 at 8:53 PM, Bejoy Ks <span dir=
=3D"ltr">&lt;<a href=3D"mailto:bejoy.hadoop@gmail.com" target=3D"_blank">be=
joy.hadoop@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">HI Abhay<div><br></div><div>The TaskTrackers=
 on which the reduce tasks are triggered is chosen in random based on the r=
educe slot availability. So if you don&#39;t need the reduce tasks to be sc=
heduled on some=A0particular=A0nodes you need to set &#39;mapred.tasktracke=
r.reduce.tasks.maximum&#39; on those nodes to 0. The bottleneck here is tha=
t this=A0property=A0is not a job level one you need to set it on a cluster =
level.</div>

<div><br></div><div>A cleaner approach will be to configure each of your no=
des with the right number of map and reduce slots based on the resources=A0=
available=A0on each machine.<div><div class=3D"h5"><br><br><div class=3D"gm=
ail_quote">
On Mon, Sep 3, 2012 at 7:49 PM, Abhay Ratnaparkhi <span dir=3D"ltr">&lt;<a =
href=3D"mailto:abhay.ratnaparkhi@gmail.com" target=3D"_blank">abhay.ratnapa=
rkhi@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello,<div><br></div><div>How can one get to=
 know the nodes on which reduce tasks will run?</div><div><br></div><div>On=
e of my job is running and it&#39;s completing all the map tasks.</div>

<div>My map tasks write lots of intermediate data. The intermediate directo=
ry is getting full on all the nodes.=A0</div>
<div>If the reduce task take any node from cluster then It&#39;ll try to co=
py the data to same disk and it&#39;ll eventually fail due to Disk space re=
lated exceptions.</div><div><br></div><div>I have added few more tasktracke=
r nodes in the cluster and now want to run reducer on new nodes only.</div>


<div>Is it possible to choose a node on which the reducer will run? What=
9;s the algorithm hadoop uses to get a new node to run reducer?</div><div><=
br></div><div>Thanks in advance.</div><div><br></div><div>Bye</div><span><f=
ont color=3D"#888888"><div>


Abhay</div>
</font></span></blockquote></div><br></div></div></div>
</blockquote></div><br></div>

--047d7b343958d7df4704c8cde87f--