hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From biro lehel <lehel.b...@yahoo.com>
Subject Transfer archives (or any file) from Mapper to Reducer?
Date Mon, 21 May 2012 08:12:18 GMT
Dear all,

In my Mapper, I run a script that processes my set of input text files, creates from them
some other text files (this is done locally on the FS on my nodes), and as a result, each
MapTask will produce an archive as a result. My issue is, that I'm looking for a way for the
Reducer to "take" these archives as some kind of an input. I understood that the communication
between Mapper-Reducer is done through the means of the key-value pairs in the Context, but
what I would need is the transferring of these archive files to the respective Reducer (I
would probably have one single Reducer, so all the files should be transferred/copied there

Is this possible? Is there a way to transfer files from Mapper to Reducer? If not, what is
the best approach in scenarios like mine? Any suggestions would be greatly appreciated.

Thank you in advance,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message