hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: How to Influence Reduce Task Location.
Date Mon, 20 Dec 2010 03:26:53 GMT
On 12/18/2010 12:43 PM, Jane Chen wrote:
> Hi All,
>
> Is there anyway to influence where a reduce task is run?  We have a case where we'd like
to choose the host to run the reduce task based on the task's input key.
>
> Any suggestion is greatly appreciated.
>
> Thanks,
> Jane

We don't do exactly that, but we do something similar.

We don't make specific reducers run on specific hosts.  But we do 
specifically shard our data - e.g., into 1024 shards - and we then run 
1024 reducers, each of which runs on its correspondingly numbered shard 
of the data.

DR

Mime
View raw message