hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1289) PIG Join fails while doing a filter on joined data
Date Wed, 17 Mar 2010 00:07:27 GMT

    [ https://issues.apache.org/jira/browse/PIG-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846222#action_12846222
] 

Daniel Dai commented on PIG-1289:
---------------------------------

Yes, it is safe not push filter up a branch that will be producing nulls. I might be wrong
but what I did is try to be a little bit more aggressive. Since the only extra value outer
join will produce is null, so if filter is not testing null, we can still push it up even
if it is on the inner branch. 

Eg:
A = load 'foo' as (q, r, s);
B = load 'bar ' as (t, u, v);
C = join A on q outer, B on t;
D = filter C by t > 0;

The production C consists of two parts:
A + B
A + "null"

If we do a filter after join, it is a union on this two parts:
filter(A + B) union filter(A + "null")

If we are not testing nullability (eg, t > 0), then filter(A + "null") will not have any
production, so
filter(A + B) union filter(A + "null") = filter(A + B)

In this case, outer join is equivalent as a regular join (since all generated null B records
are filtered away), so we can still push the filter up.

> PIG Join fails while doing a filter on joined data
> --------------------------------------------------
>
>                 Key: PIG-1289
>                 URL: https://issues.apache.org/jira/browse/PIG-1289
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Karim Saadah
>            Assignee: Daniel Dai
>            Priority: Minor
>             Fix For: 0.7.0
>
>         Attachments: PIG-1289-1.patch
>
>
> PIG Join fails while doing a filter on joined data
> Here are the steps to reproduce it:
> -bash-3.1$ pig -latest -x local
> grunt> a = load 'first.dat' using PigStorage('\u0001') as (f1:int, f2:chararray);
> grunt> DUMP a;
> (1,A)
> (2,B)
> (3,C)
> (4,D)
> grunt> b = load 'second.dat' using PigStorage() as (f3:chararray);
> grunt> DUMP b;
> (A)
> (D)
> (E)
> grunt> c = join a by f2 LEFT OUTER, b by f3;
> grunt> DUMP c;
> (1,A,A)
> (2,B,)
> (3,C,)
> (4,D,D)
> grunt> describe c;
> c: {a::f1: int,a::f2: chararray,b::f3: chararray}
> grunt> d = filter c by (f3 is null or f3 =='');
> grunt> dump d;
> 2010-03-03 15:00:37,129 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No column pruned for b
> 2010-03-03 15:00:37,129 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No map keys pruned for b
> 2010-03-03 15:00:37,129 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No column pruned for a
> 2010-03-03 15:00:37,130 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns
- No map keys pruned for a
> 2010-03-03 15:00:37,130 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1002: Unable
to store alias d
> This one is failing too:
> grunt> d = filter c by (b::f3 is null or b::f3 =='');
> or this one not returning results as expected:
> grunt> d = foreach c generate f1 as f1, f2 as f2, f3 as f3;
> grunt> e = filter d by (f3 is null or f3 =='');
> grunt> DUMP e;
> (1,A,)
> (2,B,)
> (3,C,)
> (4,D,)
> while the expected result is
> (2,B,)
> (3,C,)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message