pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-858) Order By followed by "replicated" join fails while compiling MR-plan from physical plan
Date Wed, 14 Oct 2009 20:30:31 GMT

    [ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765735#action_12765735
] 

Ashutosh Chauhan commented on PIG-858:
--------------------------------------

Its been a while since I did that patch. So, bit more clarification: We are interested in
finding PO which corresponds to "fragment" PO input of POFRJoin. This PO is already compiled
and is in one the MROper. Earlier we  will iterate through compiledInputs array trying to
match this PO  with PO contained in each MROperator. This fails as discussed in previous comments.
With this change, since we keep track of MR operator with each physical operator it need not
to do that but can simply look up for MROper corresponding to "fragment" PO in the phyToMROpMap.

> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
>                 Key: PIG-858
>                 URL: https://issues.apache.org/jira/browse/PIG-858
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.4.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.6.0
>
>         Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling
operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException:
ERROR 2034: Error compiling operator POFRJoin
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
>         at org.apache.pig.PigServer.explain(PigServer.java:574)
>         ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException:
ERROR 2034: Error compiling operator POFRJoin
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
>         ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
>         ... 16 more
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message