hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Reducer MapFileOutpuFormat
Date Fri, 27 Jul 2012 22:06:28 GMT
Hey Mike,


On Tue, Jul 24, 2012 at 1:39 AM, Mike S <mikesam460@gmail.com> wrote:
> If I set my reducer output to map file output format and the job would
> say have 100 reducers, will the output generate 100 different index
> file (one for each reducer) or one index file for all the reducers
> (basically one index file per job)?

Each MapFile gets its own index file, so yes 100 index files for 100
map files, given each Reducer creating one map file as its output.

> If it is one index file per reducer, can rely on HDFS append to change
> the index write behavior and build one index file from all the
> reducers by basically making all the parallel reducers to append to
> one index file? Data files do not matter.

I don't think MapFiles are append-able yet. For one, the index
complicates things as it keeps offsets based from 0 to length (I
think). Work for easy-appending sequence files has been ongoing
though: https://issues.apache.org/jira/browse/HADOOP-7139. Maybe you
can take a look and help enable MapFiles do the same somehow?

Harsh J

View raw message