Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABVuHihtxgiUezMmauZoV_WRusbguMkGjN2geUObA6xvP4P6Uw@mail.gmail.com>
References: 
 <CABVuHihtxgiUezMmauZoV_WRusbguMkGjN2geUObA6xvP4P6Uw@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 7 Nov 2012 18:11:05 +0530
Message-ID: 
 <CAOcnVr39PrK1=jk-WK6HgJBJM17pWsNfgmq-SV3acBusYYG3LA@mail.gmail.com>
Subject: Re: Spill file compression
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Yes we do compress each spill output using the same codec as specified
for map (intermediate) output compression. However, the counted bytes
may be counting decompressed values of the records written, and not
post-compressed ones.

On Wed, Nov 7, 2012 at 6:02 PM, Sigurd Spieckermann
<sigurd.spieckermann@gmail.com> wrote:
> Hi guys,
>
> I've encountered a situation where the ratio between "Map output bytes" and
> "Map output materialized bytes" is quite huge and during the map-phase data
> is spilled to disk quite a lot. This is something I'll try to optimize, but
> I'm wondering if the spill files are compressed at all. I set
> mapred.compress.map.output=true and
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
> and everything else seems to be working correctly. Does Hadoop actually
> compress spills or just the final spill after finishing the entire map-task?
>
> Thanks,
> Sigurd


-- 
Harsh J