hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <lordjoe2...@gmail.com>
Subject Does anyone have sample code for forcing a custom InputFormat to use a small split
Date Mon, 12 Sep 2011 02:27:40 GMT
I have a problem where there is a single, relatively small (10-20 MB) input
file. (It happens it is a fasta file which will have meaning if you are a
biologist.)  I am already using a custom  InputFormat  and a custom reader
to force a custom parsing. The file may generate tens or hundreds of
millions of key value pairs and the mapper does a fair amount of work on
each record.
The standard implementation of
 * public List<InputSplit> getSplits(JobContext job) throws IOException {*

uses fs.getFileBlockLocations(file, 0, length); to determine the blocks and
for a file of this size will come up with a single InputSplit and a single

I am looking for a good example of forcing the generation of multiple
InputSplits for a small file. In this case I am  happy if every Mapper
instance is required to read and parse the entire file    as long as I can
guarantee that every record is processed by only a single mapper.
While I think I see how I might modify*  getSplits(JobContext job)  *I am
not sure how and when the code is called when the job is running on the

Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com

View raw message