hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Question about how Hadoop stores intermediate results
Date Sun, 25 Sep 2011 19:55:03 GMT
There is only one file per-map. Actually two, an output file and an index file to quickly get
the offset/length for a given reducer.

The index file is also cached in memory for performance.

Arun

On Sep 25, 2011, at 10:00 AM, He Chen wrote:

> Hi everyone
> 
> According to my understanding of Hadoop, it save MapReduce  job's
> intermediate results into files in the mapper's hard drive. Each key will
> occupy a file. I am curious what will happen if mapper's hard drive does not
> have enough inodes to save the generated keys.  Because every file needs a
> inode.
> 
> Best wishes!
> 
> Chen


Mime
View raw message