Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Message-ID: <1548622642.1254769951593.JavaMail.jira@brutus>
Date: Mon, 5 Oct 2009 12:12:31 -0700 (PDT)
From: "Richard Ding (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Subject: [jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group
 operator to speed up processing of ordered data
In-Reply-To: <1177352385.1254337763474.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Ding updated PIG-984:
-----------------------------

    Assignee: Richard Ding

> PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data 
> ----------------------------------------------------------------------------------------
>
>                 Key: PIG-984
>                 URL: https://issues.apache.org/jira/browse/PIG-984
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Richard Ding
>            Assignee: Richard Ding
>
> The general group by operation in Pig needs both mappers and reducers (the aggregation is done in reducers). This incurs disk writes/reads  between mappers and reducers.
> However, in the cases where the input data has the following properties
>    1. The records with the same key are grouped together (such as the data is sorted by the keys).
>    2. The records with the same key are in the same mapper input.
> the group by operation can be performed in the mappers only and thus remove the overhead of disk writes/reads.
> Alan proposed adding a hint to the group by clause like this one:
> {code}
> A = load 'input' using SomeLoader(...);
> B = group A by $0 using "mapside";
> C = foreach B generate ...
> {code}
> The proposed addition of using "mapside" to group will be a mapside group operator that collects all records for a given key into a buffer. When it sees a key change it will emit the key and bag for records it had buffered. It will assume that all keys for a given record are collected together and thus there is not need to buffer across keys. 
> It is expected that "SomeLoader" will be implemented by data systems such as Zebra to ensure the data emitted by the loader satisfies the above properties (1) and (2).
> It will be the responsibility of the user (or the loader) to guarantee these properties (1) & (2) before invoking the mapside hint for the group by clause. The Pig runtime can't check for the errors in the input data.
> For the group by clauses with mapside hint, Pig Latin will only support group by columns (including *), not group by expressions nor group all. 
>   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.