hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: 2 Reduce method in one Job
Date Sun, 24 Mar 2013 13:44:11 GMT
You seem to want to re-sort/partition your data without materializing
it onto HDFS.

Azuryy is right: There isn't a way right now and a second job (with an
identity mapper) is necessary. With YARN this is more possible to
implement into the project, though.

The newly inducted incubator project Tez sorta targets this. Its in
its nascent stages though (for general user use), and the website
should hopefully appear at
http://incubator.apache.org/projects/tez.html soon. Meanwhile, you can
read the proposal behind this project at
http://wiki.apache.org/incubator/TezProposal. Initial sources are at
https://svn.apache.org/repos/asf/incubator/tez/trunk/.

On Sun, Mar 24, 2013 at 6:33 PM, Fatih Haltas <fatih.haltas@nyu.edu> wrote:
> I want to get reduce output as key and value then I want to pass them to a
> new reduce as input key and input value.
>
> So is there any Map-Reduce-Reduce kind of method?
>
> Thanks to all.



-- 
Harsh J

Mime
View raw message