hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-467) compressing intermediate results
Date Tue, 30 Sep 2008 00:16:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635619#action_12635619

Arun C Murthy commented on PIG-467:

It definitely is better to use lzo rather than zlib for compressing intermediate map-outputs.
However, lzo might not be available on the cluster, and the right 32/64 bit lzo libraries
need to installed. Hence, it would be pertinent to default to zlib but have an easy way for
the user or admin to change it to lzo.

> compressing intermediate results
> --------------------------------
>                 Key: PIG-467
>                 URL: https://issues.apache.org/jira/browse/PIG-467
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>             Fix For: types_branch
> It is recommended with Hadoop 18 and later versions to compress data passed between Map
and Reduce. We need to test to make sure that it gives performance gain

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message