hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich" <ol...@yahoo-inc.com>
Subject RE: Is it possible to access Configuration in UDF ?
Date Tue, 04 Aug 2009 14:35:33 GMT
At the moment we can't make UDFs dependant on Hadoop as people also use
them for testing in local mode which is currently not based on Hadoop
local mode due to performance constrains.

I agree that we need to provide a way to get UDF a
configuration/property object.

Olga

-----Original Message-----
From: Daniel Dai [mailto:daijyc@gmail.com] 
Sent: Monday, August 03, 2009 9:20 PM
To: pig-dev@hadoop.apache.org; pig-user@hadoop.apache.org
Subject: Re: Is it possible to access Configuration in UDF ?

Hi, Jeff,
This is not API at all, this is a hack to make things work. We do lack 
couples of features for UDF:
1. reporter and counter (PIG-889)
2. access global properties
3. ability to maintain states across different UDF invocations
4. input schema
5. variable length arguments (PIG-902)

Your suggestion sounds resonable. We need to provide a well designed 
interface for these features.

----- Original Message ----- 
From: "zhang jianfeng" <zjffdu@gmail.com>
To: <pig-user@hadoop.apache.org>; <pig-dev@hadoop.apache.org>
Sent: Monday, August 03, 2009 8:03 PM
Subject: Re: Is it possible to access Configuration in UDF ?


> Dmitriy,
>
> Thank you for your help.
>
> I find this way of using API is not so intuitive ,  I recommend the
base
> class of UDF to implements the Configurable interface.
> Then each UDF can use the getConf() to get the Configuration object.
> Because UDF is part of MapReduce , it makes sense to make it
Configurable.
>
> The following is what I recommend to change the EvalFunc
>
> public abstract class EvalFunc<T>  implements Configurable{
> ......
> protected Configuration conf;
> ......
> public EvalFunc(){
>         conf=PigMapReduce.sJobConf;
> }
> ......
>     @Override
>    public void setConf(Configuration conf) {
>        this.conf=conf;
>    }
>
>    @Override
>    public Configuration getConf() {
>        return this.conf;
>    }
>
>
>
>
> Jeff Zhang
>
>
>
>
>
> On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy 
> <dvryaboy@cloudera.com>wrote:
>
>> You can access the JobConf with the following call:
>>
>> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>>
>> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zjffdu@gmail.com>
wrote:
>> > Hi all,
>> >
>> > I'd like to set property in Configuration to customize my UDF. But
it
>> looks
>> > like I can not access the Configuration object in UDF.
>> >
>> > Does pig have a plan to support this feature ?
>> >
>> >
>> > Thank you.
>> >
>> > Jeff Zhang
>> >
>>
> 


Mime
View raw message