hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Viraj Bhat (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-568) Reducer plan generation fails when UDF contains integers as parameters
Date Wed, 17 Dec 2008 02:24:44 GMT

     [ https://issues.apache.org/jira/browse/PIG-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Viraj Bhat updated PIG-568:
---------------------------

    Attachment: myudfint.pig

Pig Script causing the exception

> Reducer plan generation fails when UDF contains integers as parameters
> ----------------------------------------------------------------------
>
>                 Key: PIG-568
>                 URL: https://issues.apache.org/jira/browse/PIG-568
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Viraj Bhat
>            Priority: Critical
>             Fix For: types_branch
>
>         Attachments: myudfint.pig, visits.txt
>
>
> Consider the following pig script which contains a UDF known as MYUDF. MYUDF is a dummy
UDF which takes in a Bag E and a set of integers as offsets. 
> {code}
> register myudf.jar;
> A = load 'visits.txt' using PigStorage() as ( name:chararray, url:chararray, timestamp:chararray);
> B = filter A by (
>         (name is not null) AND
>         (timestamp is not null)
>         );
> C = group B by (
>         url
>         );
> D = foreach C {
>         E = order B by timestamp;
>         generate E;
>         }
> G = foreach D generate
>         param.MYUDF(E, -1, 0, 1);
> --this works
> --param.MYUDF(E,'-1','0','1'); 
> explain G;
> dump G;
> {code}
> If you execute the above script, it fails during the reducer phase where the POUserFunc(MYUDF)[bag]
is being called. The MYUDF code is infact not called but somehow the parameters passed to
the MYUDF cause the exception in the reduce plan. If you replace the offsets -1,0,1 with '-1',
'0', '1' (strings) the UDF seems to get called and the script works fine. 
> =============================================================================================================================
> java.io.IOException: Received Error while processing the reduce plan.
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:307)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:247)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:224)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:136)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
> =============================================================================================================================

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message