hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <bejoy.had...@gmail.com>
Subject Re: knowing the nodes on which reduce tasks will run
Date Mon, 03 Sep 2012 15:23:11 GMT
HI Abhay

The TaskTrackers on which the reduce tasks are triggered is chosen in
random based on the reduce slot availability. So if you don't need the
reduce tasks to be scheduled on some particular nodes you need to set
'mapred.tasktracker.reduce.tasks.maximum' on those nodes to 0. The
bottleneck here is that this property is not a job level one you need to
set it on a cluster level.

A cleaner approach will be to configure each of your nodes with the right
number of map and reduce slots based on the resources available on each

On Mon, Sep 3, 2012 at 7:49 PM, Abhay Ratnaparkhi <
abhay.ratnaparkhi@gmail.com> wrote:

> Hello,
> How can one get to know the nodes on which reduce tasks will run?
> One of my job is running and it's completing all the map tasks.
> My map tasks write lots of intermediate data. The intermediate directory
> is getting full on all the nodes.
> If the reduce task take any node from cluster then It'll try to copy the
> data to same disk and it'll eventually fail due to Disk space related
> exceptions.
> I have added few more tasktracker nodes in the cluster and now want to run
> reducer on new nodes only.
> Is it possible to choose a node on which the reducer will run? What's the
> algorithm hadoop uses to get a new node to run reducer?
> Thanks in advance.
> Bye
> Abhay

View raw message