hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: I want to group "similar" keys in the reducer.
Date Mon, 15 Mar 2010 18:20:00 GMT
you need to define a pattern and implement you own partitioner so that all the similar keys
you want to group will go the the same reducer. At reduce side, you possibly need to  implement
secondary  sorting so that the keys you want to group are grouped in the sorted input to reducer.
For reduce method process on key at one time, you also need to maintain a window to buffer
all the keys being grouped.

-Gang



----- 原始邮件 ----
发件人: Raymond Jennings III <raymondjiii@yahoo.com>
收件人: common-user@hadoop.apache.org
发送日期: 2010/3/15 (周一) 1:26:09 下午
主   题: I want to group "similar" keys in the reducer.

Is it possible to override a method in the reducer so that similar keys will be grouped together?
 For example I want all keys of value "KEY1" and "KEY2" to merged together.  (My reducer has
a KEY of type TEXT.)  Thanks.


      

Mime
View raw message