hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Olston (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-552) UDF defined with argument causes class instantiation exception
Date Wed, 03 Dec 2008 21:01:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652971#action_12652971
] 

Christopher Olston commented on PIG-552:
----------------------------------------

No, that's not it. Pig tries to pass the string "myFunc('blah')" to the class loader, which
of course fails. This seems to be some sort of test for whether it's a comparator function,
which is not the case anyway -- it's an EvalFunc -- so the test is supposed to fail, but not
throw an exception from the class loader. My patch fixes this.

I cannot post the UDF -- it's confidential.

> UDF defined with argument causes class instantiation exception
> --------------------------------------------------------------
>
>                 Key: PIG-552
>                 URL: https://issues.apache.org/jira/browse/PIG-552
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Christopher Olston
>         Attachments: pig.patch
>
>
> I'm doing:
> define myFunc myFunc('blah');
> b = foreach a generate myFunc(*);
> Pig parses it, but fails when it tries to run it on hadoop (I'm using "local" mode).
It tries to invoke the class loader on "myFunc('blah')" instead of on "myFunc", which causes
an exception.
> The bug seems to stem from this part of JobControlCompiler.getJobConf():
>                 if(mro.UDFs.size()==1){
>                     String compFuncSpec = mro.UDFs.get(0);
>                     Class comparator = PigContext.resolveClassName(compFuncSpec);
>                     if(ComparisonFunc.class.isAssignableFrom(comparator)) {
>                         jobConf.setMapperClass(PigMapReduce.MapWithComparator.class);
>                         jobConf.setReducerClass(PigMapReduce.ReduceWithComparator.class);
>                         jobConf.set("pig.reduce.package", ObjectSerializer.serialize(pack));
>                         jobConf.set("pig.usercomparator", "true");
>                         jobConf.setOutputKeyClass(NullableTuple.class);
>                         jobConf.setOutputKeyComparatorClass(comparator);
>                     }
>                 } else {
>                     jobConf.set("pig.sortOrder",
>                         ObjectSerializer.serialize(mro.getSortOrder()));
>                 }

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message