hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ondřej Klimpera <klimp...@fit.cvut.cz>
Subject Working with MapFiles
Date Thu, 29 Mar 2012 15:05:25 GMT

I have a MapFile as a product of MapReduce job, and what I need to do is:

1. If MapReduce produced more spilts as Output, merge them to single file.

2. Copy this merged MapFile to another HDFS location and use it as a 
Distributed cache file for another MapReduce job.

I'm wondering if it is even possible to merge MapFiles according to 
their nature and use them as Distributed cache file.

What I'm trying to achieve is repeatedly fast search in this file during 
another MapReduce job.
If my idea is absolute wrong, can you give me any tip how to do it?

The file is supposed to be 20MB large.
I'm using Hadoop 0.20.203.

Thanks for your reply:)

Ondrej Klimpera

View raw message