hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marek Miglinski <mmiglin...@seven.com>
Subject codec compression ratio
Date Thu, 14 Jun 2012 08:39:12 GMT
When procession 65billion records and using LZO or Snappy codecs, disk IO is at 100% because
mappers are spilling all the time, but CPU is at 40%. Is there a setting where I can raise
compression ratio for map/reduce internal temp data (for LZO or Snappy)? So that I can raise
effort on CPU and lower IO? Google didn't gave any ideas...


Thanks.
Marek M.

Mime
View raw message