hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-467) compressing intermediate results
Date Tue, 30 Sep 2008 00:16:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635619#action_12635619
] 

Arun C Murthy commented on PIG-467:
-----------------------------------

It definitely is better to use lzo rather than zlib for compressing intermediate map-outputs.
However, lzo might not be available on the cluster, and the right 32/64 bit lzo libraries
need to installed. Hence, it would be pertinent to default to zlib but have an easy way for
the user or admin to change it to lzo.

> compressing intermediate results
> --------------------------------
>
>                 Key: PIG-467
>                 URL: https://issues.apache.org/jira/browse/PIG-467
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>             Fix For: types_branch
>
>
> It is recommended with Hadoop 18 and later versions to compress data passed between Map
and Reduce. We need to test to make sure that it gives performance gain

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message