hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Broberg <Tim.Brob...@exar.com>
Subject RE: codec compression ratio
Date Thu, 14 Jun 2012 15:55:34 GMT
Have you considered deflate or bzip?

    - Tim.

From: Marek Miglinski [mmiglinski@seven.com]
Sent: Thursday, June 14, 2012 1:39 AM
To: mapreduce-user@hadoop.apache.org
Subject: codec compression ratio

When procession 65billion records and using LZO or Snappy codecs, disk IO is at 100% because
mappers are spilling all the time, but CPU is at 40%. Is there a setting where I can raise
compression ratio for map/reduce internal temp data (for LZO or Snappy)? So that I can raise
effort on CPU and lower IO? Google didn't gave any ideas...

Marek M.

The information contained in this email is intended only for the personal and confidential
use of the recipient(s) named above.  The information and any attached documents contained
in this message may be Exar confidential and/or legally privileged.  If you are not the intended
recipient, you are hereby notified that any review, use, dissemination or reproduction of
this message is strictly prohibited and may be unlawful.  If you have received this communication
in error, please notify us immediately by return email and delete the original message.

View raw message