pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-5381) Extra foreach in nested-foreach preventing Accumulator usage
Date Tue, 05 Mar 2019 15:56:00 GMT

     [ https://issues.apache.org/jira/browse/PIG-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Koji Noguchi updated PIG-5381:
------------------------------
    Attachment: pig-5381-v01.patch

Uploading a patch, {{pig-5381-v01.patch}}, but not confident.
 Looking at the code, I feel that num_parameter=1 check for accumulator should be done at
{{AccumulatorOptimizerUtil.check}} level instead of {{AccumulatorOptimizerUtil.checkUDFInput}}
level.

This patch passes the test case and no regression in test failures but I must be missing something
since this part of the code has been there for years.

> Extra foreach in nested-foreach preventing Accumulator usage
> ------------------------------------------------------------
>
>                 Key: PIG-5381
>                 URL: https://issues.apache.org/jira/browse/PIG-5381
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Major
>         Attachments: pig-5381-v01.patch
>
>
> {code}
> A = LOAD 'test.input1' AS (a1:int, a2:chararray, a3:int);
> B = GROUP A by a1;
> C = FOREACH B {
>     C2 = FOREACH A generate a2, a3;
>     GENERATE group, org.apache.pig.test.utils.AccumulatorBagCount(C2);
> }
> dump C;
> {code} 
> This simple code fails with 
> {noformat}
> Caused by: java.io.IOException: exec() should not be called.
>         at org.apache.pig.test.utils.AccumulatorBagCount.exec(AccumulatorBagCount.java:56)
>         at org.apache.pig.test.utils.AccumulatorBagCount.exec(AccumulatorBagCount.java:28)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:332)
>         ... 17 more
> {noformat}
> Somehow pig decides not to enable Accumulator in this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message