hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-425) Split -> distinct or order -> cogroup fails
Date Tue, 09 Sep 2008 19:13:44 GMT
Split -> distinct or order -> cogroup fails
-------------------------------------------

                 Key: PIG-425
                 URL: https://issues.apache.org/jira/browse/PIG-425
             Project: Pig
          Issue Type: Bug
          Components: impl
    Affects Versions: types_branch
            Reporter: Alan Gates
            Assignee: Alan Gates
             Fix For: types_branch


A script like:

{code}
\a = load 'myfile' as (name:chararray, age:int, gpa:double);
split a into a1 if age > 50, a2 if name < 'm';
b2 = distinct a2;
b1 = order a1 by name;
c = cogroup b2 by name, b1 by name;
d = foreach c generate flatten(group), COUNT($1), COUNT($2);
store d into 'OUTPATH';
{code}

Will abort with the error:
{code}
08/09/09 11:46:50 ERROR mapReduceLayer.Launcher: Error message from task (map) tip_200809080906_0185_m_000000java.lang.ClassCastException:
org.apache.pig.data.DefaultTuple cannot be cast to org.apache.pig.data.IndexedTuple
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:81)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:135)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
{code}

The issue is that the RearrangeAdjuster in MRCompiler is not properly seeing this as a cogroup
and moving the localrearrnge out of the reduce and into the
map.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message