hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Map output files are SequenceFileFormat
Date Mon, 14 Feb 2011 18:16:52 GMT

On Mon, Feb 14, 2011 at 11:37 PM, Pedro Costa <psdc1978@gmail.com> wrote:
> And when the data of the map-intermediate files is compressed, it's
> still an IFile?

Yes. From my understanding, if compression is turned ON for IFile, the
output stream for writing the IFile is itself set as a compressing one
and all data written to the stream is compressed.

In contrast, in SequenceFiles, compression is done in blocks (of a
sizes set upon the Writer creation), and keys are left uncompressed.

Harsh J

View raw message