hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2402) Lzo compression compresses each write from TextOutputFormat
Date Thu, 13 Dec 2007 00:55:43 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Douglas updated HADOOP-2402:

    Attachment: 2402-0.patch

Inserted a 64k BufferedOutputStream to TextOutputFormat::getRecordWriter. Effects compression
similar to block, lzo compressed SequenceFile (~20%). For comparison, lzop (command line compression
utility backed by lzo lib) uses 256k blocks (and a different file format) and compresses the
same 100M sample by 60%.

As an aside, with this applied, Zip compressed text files are written approximately 10% faster;
uncompressed text about 4% faster.

> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>                 Key: HADOOP-2402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2402
>             Project: Hadoop
>          Issue Type: Bug
>          Components: io, mapred, native
>            Reporter: Chris Douglas
>             Fix For: 0.16.0
>         Attachments: 2402-0.patch
> Outputting with TextOutputFormat and Lzo compression generates a file such that each
key, tab delimiter, and value are compressed separately.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message