hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-43) Combiner queries are not repeatable using the same alias
Date Thu, 06 Dec 2007 19:02:43 GMT

    [ https://issues.apache.org/jira/browse/PIG-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549134
] 

Olga Natkovich commented on PIG-43:
-----------------------------------

looks good. +1

> Combiner queries are not repeatable using the same alias
> --------------------------------------------------------
>
>                 Key: PIG-43
>                 URL: https://issues.apache.org/jira/browse/PIG-43
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.0.0
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Minor
>         Attachments: pig43.patch
>
>
> In general, it is possible to define an alias in pig and use it multiple times:
> a = load 'file';
> b = group a by $0;
> c = foreach b generate group, COUNT($1);
> dump c;
> dump c;
> Queries that choose to use the combiner give the following error on the second dump:
> java.lang.AssertionError: Can't convert non-algebraic function to final.
>         at org.apache.pig.impl.eval.FuncEvalSpec.resetFuncToFinal(FuncEvalSpec.java:285)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler$ReduceAdjuster.visitFuncEval(MapreducePlanCompiler.java:393)
>         at org.apache.pig.impl.eval.FuncEvalSpec.visit(FuncEvalSpec.java:238)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler$ReduceAdjuster.visitGenerate(MapreducePlanCompiler.java:367)
>         at org.apache.pig.impl.eval.GenerateSpec.visit(GenerateSpec.java:374)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler.pushInto(MapreducePlanCompiler.java:223)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler.compile(MapreducePlanCompiler.java:79)
>         at org.apache.pig.impl.physicalLayer.PhysicalPlan.<init>(PhysicalPlan.java:56)
>         at org.apache.pig.impl.physicalLayer.IntermedResult.compile(IntermedResult.java:95)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:319)
>         at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:120)
>         at org.apache.pig.tools.grunt.GruntParser.parse(GruntParser.java:334)
>         at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:38)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:48)
>         at org.apache.pig.Main.main(Main.java:211)
> This is because the logic that chooses the combiner modifies the EvalSpecs in the logical
plan.  Pig does not preserve the physical plan built for the first run of the query, but tries
to reconstruct it. When it does so the modified EvalSpecs cause it a problem.  The combiner
logic needs to be changed to make a copy of the EvalSpecs rather than modify them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message