pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai" <dai...@gmail.com>
Subject Re: Review Request: Bug in new logical plan : No output generated even though there are valid records
Date Fri, 18 Mar 2011 21:34:17 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated 2011-03-18 14:34:17.932065)

Review request for pig and thejas.


I have the below script which provides me no output even though there are valid records in
relation B which is used for the left out join.

A0 = load 'input' using Maploader() as ( map1, map2, map3 );
A = filter A0 by ( (map2#'params'#'prop' == 464) and (map2#'params'#'query' is not null) );
B0 = filter A by (map1#'type' == 'c');
B = filter B0 by ( map2#'info'#'s' matches 'aaaa|bbb|cccc');
C = filter A by (map1#'type' == 'p');
D = join B by map2#'params'#'query' LEFT OUTER , C by map2#'params'#'query';
store D into 'output';

This is a bug with the newlogical plan. From the plan i can see that map1#'type' and map2#'info'#'s'
is not marked as RequiredKeys ,
but where as all the fields reffered in the firts filter statement is marked as required.

For the script to work I have to turn off the coloumn prune optimizer by -t ColumnMapKeyPrune
or rearrange the script such that;
B0 = filter A0 by ( (map2#'params'#'prop' == 464) and (map2#'params'#'query' is not null)
and (map1#'type' == 'c') );
C = filter A0 by ( (map2#'params'#'prop' == 464) and (map2#'params'#'query' is not null) and
(map1#'type' == 'p') );

This addresses bug PIG-1892.



Diff: https://reviews.apache.org/r/517/diff

Testing (updated)

     [exec] -1 overall.  
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]     -1 release audit.  The applied patch generated 541 release audit warnings
(more than the trunk's current 539 warnings).
No new files added, ignore "release audit warnings".

Unit test:
    all pass



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message