hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (PIG-710) Filtering bag in nested foreach does not produce expected results
Date Thu, 16 Apr 2009 18:38:15 GMT

     [ https://issues.apache.org/jira/browse/PIG-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pradeep Kamath resolved PIG-710.
--------------------------------

       Resolution: Duplicate
    Fix Version/s: 0.3.0
         Assignee: Pradeep Kamath

This issue has the same root cause as PIG-514 - hence marking this duplicate - the fix for
this issue will also be tracked in PIG-514

> Filtering bag in nested foreach does not produce expected results
> -----------------------------------------------------------------
>
>                 Key: PIG-710
>                 URL: https://issues.apache.org/jira/browse/PIG-710
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>            Assignee: Pradeep Kamath
>             Fix For: 0.3.0
>
>
> I have an idiom I used to use in older versions of pig (prior to types branch) which
would group into a collection and then filter the output if any of the collection contained
a particular string.
> This relies on FILTER statements within a FOREACH ... { ... GENERATE ... } statement.
> ORDER ... BY in the FOREACH ... { ... GENERATE ... } statement does not seem to have
a problem so it seems to be something isolated to the FILTER.
> {code}
> A = load 'filterbug.data' using PigStorage() as ( id, str );
> B = group A by ( id );
> describe B;
> dump B;
> D = foreach B generate
>         group,
>         COUNT(A),
>         A.str;
> describe D;
> dump D;
> C = foreach B {
>         D = order A by str;
>         matchedcount = COUNT(D);
>         generate
>                 group,
>                 matchedcount as matchedcount,
>                 D.str;
>         };
> describe C;
> dump C;
> Cfiltered = foreach B {
>         D = filter A by (
>                 str matches 'hello'
>                 );
>         matchedcount = COUNT(D);
>         generate
>                 group,
>                 matchedcount as matchedcount,
>                 A.str;
>         };
> describe Cfiltered;
> dump Cfiltered;
> {code}
> Here's the output:
> {code}
> -bash-3.00$ pig -exectype local -latest filterbug.pig
> USING: /grid/0/gs/pig/current
> B: {group: bytearray,A: {id: bytearray,str: bytearray}}
> 2009-03-10 03:14:14,838 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- 100% complete!
> 2009-03-10 03:14:14,839 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- Success!!
> (a,{(a,hello),(a,goodbye)})
> (b,{(b,goodbye)})
> (c,{(c,hello),(c,hello),(c,hello)})
> (d,{(d,what)})
> D: {group: bytearray,long,str: {str: bytearray}}
> 2009-03-10 03:14:14,920 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- 100% complete!
> 2009-03-10 03:14:14,920 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- Success!!
> (a,2L,{(hello),(goodbye)})
> (b,1L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,1L,{(what)})
> C: {group: bytearray,matchedcount: long,str: {str: bytearray}}
> 2009-03-10 03:14:14,985 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- 100% complete!
> 2009-03-10 03:14:14,985 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- Success!!
> (a,2L,{(goodbye),(hello)})
> (b,1L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,1L,{(what)})
> 2009-03-10 03:14:15,018 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY
1 time(s).
> Cfiltered: {group: bytearray,matchedcount: long,str: {str: bytearray}}
> 2009-03-10 03:14:15,044 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY
1 time(s).
> 2009-03-10 03:14:15,057 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- 100% complete!
> 2009-03-10 03:14:15,057 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher
- Success!!
> (a,1L,{(hello),(goodbye)})
> {code}
> What I expect for the output of Cfiltered is actually:
> (a,1L,{(hello),(goodbye)})
> (b,0L,{(goodbye)})
> (c,3L,{(hello),(hello),(hello)})
> (d,0L,{(what)})
> The data file is:
> {code}
> a       hello
> a       goodbye
> b       goodbye
> c       hello
> c       hello
> c       hello
> d       what
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message