hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-490) Combiner not used when group elements referred to in tuple notation instead of flatten.
Date Fri, 10 Oct 2008 21:10:44 GMT
Combiner not used when group elements referred to in tuple notation instead of flatten.
---------------------------------------------------------------------------------------

                 Key: PIG-490
                 URL: https://issues.apache.org/jira/browse/PIG-490
             Project: Pig
          Issue Type: Bug
    Affects Versions: types_branch
            Reporter: Alan Gates
             Fix For: types_branch


Given a query like:

{code}
A = load 'myfile';
B = group A by ($0, $1);
C = foreach B generate group.$0, group.$1, COUNT(A);
{code}

The combiner will not be invoked.  But if the last line is changed to:

{code}
C = foreach B generate flatten(group), COUNT(A);
{code}

it will be.  The reason for the discrepancy is because the CombinerOptimizer checks that all
of the projections are simple.  If not, it does not use the combiner.  group.$0 is not a simple
projection, so this is failed.  However, this is a common enough case that the CombinerOptimizer
should detect it and still use the combiner. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message