hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Ding (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1458) aggregate files for replicated join
Date Thu, 26 Aug 2010 21:54:55 GMT

     [ https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Richard Ding updated PIG-1458:
------------------------------

    Attachment: PIG-1458.patch

This patch uses the new multi-file-combiner (PIG-1518) to concatenate many small files for
replicated join. This is based on the assumption that the total size of the replicated files
should be small enough to fit into main memory. 

> aggregate files for replicated join
> -----------------------------------
>
>                 Key: PIG-1458
>                 URL: https://issues.apache.org/jira/browse/PIG-1458
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Olga Natkovich
>            Assignee: Richard Ding
>             Fix For: 0.8.0
>
>         Attachments: PIG-1458.patch
>
>
> We have noticed that if the smaller data in replicated join has many files, this puts
 unneeded burden on the name node. pre-aggregating the files can improve the situation

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message