hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Injun Joe <ll_oz...@yahoo.com.hk>
Subject successive mappers
Date Fri, 15 Apr 2011 19:05:18 GMT
I am coding a map-reduce program which involves several map-reduce steps. The work that my
program does is only in the mapper, so I was thinking to have no reduce steps but successive
mappers. The logic can be written like this for mappers at iteration 0 and 1:

1. Take input.
2. Map 0:
   Determine if a key-value pair satisfies condition C. 
    - If it satisfies condition then output the key-value pair to a file in directory E.
    - If it does not then transform key-value pair and output the key-value pair to directory
3. Map 1:
   - Change input directory to directory D
   - Perform same steps as map 0.

So, the problem is that I have not been able to find a way to output key-value pairs to different
directories. All I have been able to specify is the map output directory by TextOutputFormat.setOutputPath.

Any help would be appreciated.

Thanks a lot

View raw message