hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2402) Lzo compression compresses each write from TextOutputFormat
Date Fri, 04 Jan 2008 20:20:34 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Douglas updated HADOOP-2402:
----------------------------------

    Attachment: 2402-1.patch

This patch reworks the lzo codec to fill its nio buffer before spilling to the native code.
It effects the following results for 5GB of random text:

Write:
|| Format || Type || Time (s from trunk, pcnt change) || Compression (% from uncompressed)
||
| SEQ | BLK | -51 (-31.1%) | 54.9% (23.7% using trunk) |
| SEQ | REC | +5 (+2.7%) | -3.6% (-3.6% using trunk) |
| TXT | | -96 (-63.9%) | 54.8% (-12.3% using trunk) |

Read:
|| Format || Type || Time (s from trunk, pcnt change) ||
| SEQ | BLK | +7 (+8.6%) |
| SEQ | REC | +8 (+5.6%) |
| TXT | | -83 (-39.1%) |

The default extension should probably be something other than .lzo, since we're not compatible
with lzop.

> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
>                 Key: HADOOP-2402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2402
>             Project: Hadoop
>          Issue Type: Bug
>          Components: io, mapred, native
>            Reporter: Chris Douglas
>             Fix For: 0.16.0
>
>         Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each
key, tab delimiter, and value are compressed separately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message