hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yanfeng Zhang <zhangy...@gmail.com>
Subject How to send KV pair to a reduce task on a particular machine?
Date Sat, 06 Mar 2010 03:50:26 GMT
Hi, all

The KV pairs (kv1, kv2, kv3 kv4) out from mapper would be partitioned into R
parts (e.g. R=2) by a partitioner. For example, kv1 and kv2 are in
partition1, while kv3 and kv4 are in partition2, the reducers will get KV
pairs from these two partitions, reducer1 get KV pairs from partition1 and
reducer2 get KV pairs from partition2.

I want to let machine1 get KV pairs from partition1 and machine2 get KV
pairs from partition2. But reducer1 is not always on machine1, reducer2 is
not always on machine2. Is there any way to make sure kv1 and kv2 are sent
to machine1 and kv3, kv4 are sent to machine2?

Thank you in advance!

Yanfeng Zhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message