pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Wagner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3458) ScalarExpression lost with multiquery optimization
Date Fri, 13 Sep 2013 18:47:52 GMT

    [ https://issues.apache.org/jira/browse/PIG-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766790#comment-13766790
] 

Mark Wagner commented on PIG-3458:
----------------------------------

I'd be in favor of 2 as well. There's no guarantee that the StoreFunc will also be a LoadFunc
and the scalar will be a small file so writing an extra copy isn't costly.
                
> ScalarExpression lost with multiquery optimization
> --------------------------------------------------
>
>                 Key: PIG-3458
>                 URL: https://issues.apache.org/jira/browse/PIG-3458
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>
> Our user reported an issue where their scalar results goes missing when having two store
statements.
> {noformat}
> A = load 'test1.txt' using PigStorage('\t') as (a:chararray, count:long);
> B = group A all;
> C = foreach B generate SUM(A.count) as total ;
> store C into 'deleteme6_C' using PigStorage(',');
> Z = load 'test2.txt' using PigStorage('\t') as (a:chararray, id:chararray );
> Y = group Z by id;
> X = foreach Y generate group, C.total;
> store X into 'deleteme6_X' using PigStorage(',');
> ====Inputs
>  pig> cat test1.txt
> a       1
> b       2
> c       8
> d       9
>  pig> cat test2.txt
> a       z
> b       y
> c       x
>  pig>
> {noformat}
> Result X should contain the total count of '20' but instead it's empty.
> {noformat}
>  pig> cat deleteme6_C/part-r-00000
> 20
>  pig> cat deleteme6_X/part-r-00000
> x,
> y,
> z,
>  pig>
> {noformat}
> This works if we take out first "store C" statement.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message