hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From biro lehel <lehel.b...@yahoo.com>
Subject Transfer archives (or any file) from Mapper to Reducer?
Date Mon, 21 May 2012 08:12:18 GMT
Dear all,

In my Mapper, I run a script that processes my set of input text files, creates from them
some other text files (this is done locally on the FS on my nodes), and as a result, each
MapTask will produce an archive as a result. My issue is, that I'm looking for a way for the
Reducer to "take" these archives as some kind of an input. I understood that the communication
between Mapper-Reducer is done through the means of the key-value pairs in the Context, but
what I would need is the transferring of these archive files to the respective Reducer (I
would probably have one single Reducer, so all the files should be transferred/copied there
somehow).

Is this possible? Is there a way to transfer files from Mapper to Reducer? If not, what is
the best approach in scenarios like mine? Any suggestions would be greatly appreciated.

Thank you in advance,
Lehel.




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message