pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johnny Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3329) RANK operator failed when working with SPLIT
Date Thu, 23 May 2013 17:43:23 GMT

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665400#comment-13665400
] 

Johnny Zhang commented on PIG-3329:
-----------------------------------

Redis, I actually can reproduce the issue with queries
{noformat}
d = load 'input1' as (a:int, b:int, c:int);
SPLIT d into e if a > 0, f if a < 0;
g = RANK e;
{noformat}

the error I got is from JobControlCompiler, it complains missing OperationID in MROperPlan.
I guess the reason is when compile from physical plan to MROperPlan, the operators order is
wrong, but not sure. I don't have a fix yet....

The workaround for now is using FILTER BY
{noformat}
d = load 'input1' as (a:int, b:int, c:int);
e = filter d by a > 0;
g = RANK e;
{noformat}

I post comment when I have new find. Thanks.
                
> RANK operator failed when working with SPLIT 
> ---------------------------------------------
>
>                 Key: PIG-3329
>                 URL: https://issues.apache.org/jira/browse/PIG-3329
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11.1
>            Reporter: Redis Liu
>            Priority: Critical
>
> input.txt:
> 1 2 3
> 4 5 6
> 7 8 9
> script:
> a = load 'input.txt' using PigStorage(' ') as (a:int, b:int, c:int);
> SPLIT a into b if a > 0, c if a > 5;
> d = RANK b;
> dump d;
> job will fail with error message:
> java.lang.RuntimeException: Unable to read counter pig.counters.counter_4929375455335572575_-1
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.PORank.addRank(PORank.java:161)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.PORank.getNext(PORank.java:134)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:214)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:157)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:275)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1340)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:269)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message