hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject RE: Location reduce task running.
Date Mon, 24 Aug 2009 05:25:53 GMT
No, but if you want a "reducer like" functionality on the same node, have a look at combiners.
To get exact functionality you might need to tweak around a little wrt buffers, flush etc.


From: fan wei fang [mailto:eagleeye83dp@gmail.com]
Sent: Monday, August 24, 2009 9:17 AM
To: mapreduce-user@hadoop.apache.org
Subject: Location reduce task running.

Hello guys,

I am a newbie of Hadoop and doing an experiment with Hadoop.
My situation is:
 +My job is expected to run continuously/frequently
 +My reduce task require a large amount of configuration data. This config data is specific
to map output's key.
-->That's why, I want to avoid moving this config data around.
As far as I read, nodes where reduce tasks are assigned are picked without consideration of
data locality.

My question is: Is there any way to force the reduce tasks for a specific key running on the
same node?


View raw message