hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-490) Combiner not used when group elements referred to in tuple notation instead of flatten.
Date Thu, 22 Apr 2010 06:38:50 GMT

    [ https://issues.apache.org/jira/browse/PIG-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859681#action_12859681

Scott Carey commented on PIG-490:

Ugh, so that is what is happening and why I almost never see combiners.  I almost always use
the 'generate group.field1, group.field2' notation instead of flatten because it saves me
a line projecting out "group::field1 as field1 ... " in the next line.  If I don't do the
latter it tends to fail to parse when the alias output from the FOREACH is used later.   

That is to say I avoid FLATTEN(group) because it works around other peculiarities and ends
up being harder to use than listing all the fields in the group.

This is with Pig 0.5.

> Combiner not used when group elements referred to in tuple notation instead of flatten.
> ---------------------------------------------------------------------------------------
>                 Key: PIG-490
>                 URL: https://issues.apache.org/jira/browse/PIG-490
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: Alan Gates
> Given a query like:
> {code}
> A = load 'myfile';
> B = group A by ($0, $1);
> C = foreach B generate group.$0, group.$1, COUNT(A);
> {code}
> The combiner will not be invoked.  But if the last line is changed to:
> {code}
> C = foreach B generate flatten(group), COUNT(A);
> {code}
> it will be.  The reason for the discrepancy is because the CombinerOptimizer checks that
all of the projections are simple.  If not, it does not use the combiner.  group.$0 is not
a simple projection, so this is failed.  However, this is a common enough case that the CombinerOptimizer
should detect it and still use the combiner. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message