pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3051) java.lang.IndexOutOfBoundsException failure with LimitOptimizer + ColumnPruning
Date Thu, 15 Nov 2012 18:16:13 GMT

    [ https://issues.apache.org/jira/browse/PIG-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498190#comment-13498190
] 

Koji Noguchi commented on PIG-3051:
-----------------------------------

Sorry, I missed Bill's comment.

bq. We've seen similar exceptions when loading data that contains text with the column delimiter
in it, which produces shorter than expected tuples. Could that be the case here?

I don't think that's the case here.  However, I've made many silly mistakes&misunderstandings
in pig before.  Let me double check.
                
> java.lang.IndexOutOfBoundsException  failure with LimitOptimizer + ColumnPruning
> --------------------------------------------------------------------------------
>
>                 Key: PIG-3051
>                 URL: https://issues.apache.org/jira/browse/PIG-3051
>             Project: Pig
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 0.10.0, 0.11
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>         Attachments: pig-3051-v1-withouttest.txt
>
>
> Had a user hitting 
> "Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1" error when he had
multiple stores and limit in his code.
> I couldn't reproduce this with short pig code (due to ColumnPruning somehow not happening
when shortened), but here's a snippet. 
> {noformat}
> ...
> G3 = FOREACH G2 GENERATE sortCol, FLATTEN(group) as label, (long)COUNT(G1) as cnt;
> G4 = ORDER G3 BY cnt DESC PARALLEL 25;
> ONEROW = LIMIT G4 1;
> U1 = FOREACH ONEROW GENERATE 3 as sortcol, 'somelabel' as label, cnt;
> store U1 into 'u1' using PigStorage();
> store G4 into 'g4' using PigStorage();
> {noformat}
> With '-t ColumnMapKeyPrune', job didn't hit the error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message