hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1491) Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
Date Fri, 09 Jul 2010 21:48:50 GMT

    [ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886907#action_12886907
] 

Scott Carey commented on PIG-1491:
----------------------------------

Full stack trace at the end of this comment.  

The pig script used is a couple hundred lines long.  But the individual chunk that I can change
to trigger the issue is the following:

DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
DATA = FOREACH DATA_G {
    a_items = DISTINCT A.x;
    b_items = DISTINCT B.x;
    GENERATE
    FLATTEN(group) as (a,b,c,d),
    SUM(A.m) as m,
    SUM(A.n) as n,
    COUNT(a_items) as a_item_count,
    (long)(SUM(B.u) + (double)0.5) as u,
    (long)(SUM(B.v) + (double)0.5) as v,
    COUNT(b_items) as b_item_count;
}

Removing both of the DISTINCT temporary aliases and not generating those counts works fine.
 Adding either one of them causes it to fail.





ERROR 2043: Unexpected error during execution.

org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during
execution.
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:318)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1016)
        at org.apache.pig.PigServer.execute(PigServer.java:1009)
        at org.apache.pig.PigServer.access$100(PigServer.java:114)
        at org.apache.pig.PigServer$Graph.execute(PigServer.java:1261)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:326)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:110)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:167)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
        at org.apache.pig.Main.main(Main.java:335)
Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad
cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
        at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
        at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
        at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:446)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:108)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:294)
        ... 10 more


> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
> ----------------------------------------------------------------------------------------
>
>                 Key: PIG-1491
>                 URL: https://issues.apache.org/jira/browse/PIG-1491
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Scott Carey
>
> I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.

> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad
cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
>         at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message