hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kayla Jay <kaylai...@yahoo.com>
Subject Map Intermediate key/value pairs written to file system
Date Fri, 18 Apr 2008 12:49:57 GMT
Hi

I have no reduces. I would like to directly write my map results while they are produced after
each map has completed to disk.  I don't want to collect then write to output.

If I wanted to directly write my map output 1-by-1 (intermediate key/value pairs) after each
map completes into individual files instead of collecting them until the end then writing
them in 1 swoop into the composite results file (part-000X), is this possible and how do I
do that? 

Can I force a write within the map to write the map key/value pairs as an individual file
for each results set instead of output.collect and having them all the key/value pairs written
to the output?

I.e I would like the intermediate key/value pairs produced from the maps to write to disk
immediatly than having it to collect in the end all of the key/value pairs and writing it
out.  I want individual files per key/value pair produced.

Thanks.


       
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it now.
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message