incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Ho to deploying a custom processor to demux
Date Wed, 23 Dec 2009 18:24:53 GMT
I think this was the jira that you were talking about Ari.

https://issues.apache.org/jira/browse/CHUKWA-292

Both mapper and reducer should be able to live in another name space.  The
limitation is in the current code.  CHUKWA-422 is the current plan to update
Demux for hadoop 0.20+ and we can make it more user friendly in it.

Regards,
Eric


On 12/23/09 6:43 AM, "asrabkin@gmail.com" <asrabkin@gmail.com> wrote:

> I would say that limitation is a bug. I remember opening a Jira for it, once.
> Shouldn't be too hard to fix...
> 
> sent from my iPhone; please excuse typos and brevity.
> 
> On Dec 23, 2009, at 8:36 AM, Christian <engrean@gmail.com> wrote:
> 
>> Hi Eric,
>> 
>> This isn't an issue for me yet, but I can see it becoming one if we end up
>> using Chukwa (currently investigating it). What's the reason behind requiring
>> the package name to be org.apache.hadoop.chukwa.
>> extraction.demux.processor.mapper?
>> 
>> Regards,
>> Chris
>> 
>> 
>> On Tue, Dec 22, 2009 at 6:46 PM, Eric Yang <eyang@yahoo-inc.com
>> <mailto:eyang@yahoo-inc.com> > wrote:
>>> I thought this is the current implementation.  The class file should be in
>>> the same package name, but it is not required to be in the same jar file.
>>> If it is not working, please file a jira.
>>> 
>>> Regards,
>>> Eric
>>> 
>>> 
>>> On 12/22/09 4:40 PM, "Bill Graham" <billgraham@gmail.com
>>> <mailto:billgraham@gmail.com> > wrote:
>>> 
>>>>> The extensions could be added by adding the class to the class path of
the
>>>>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>>>>> chukwa-demux-conf.xml, then it should work automatically.
>>>> 
>>>> Just to clarify, are you saying this is how it currently works or how it
>>>> could
>>>> work in the future?
>>>> 
>>>> Currently it doesn't work this way, which is the point of my post. I put
a
>>>> jar
>>>> in the lib/ directory of my data processor that contained my processor. I
>>>> mapped it in chukwa-demux.xml  and bounced the data processor. I did a ps
>>>> and
>>>> saw the jar in the DemuxManager classpath, but I still got
>>>> ClassNotFoundExceptions.
>>>> 
>>>> The only way I could get it to work was to do the following:
>>>> - Move my class into
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper
>>>> - Add my compiled class to the chukwa-core jar.
>>>> 
>>>> 
>>>> On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com
>>>> <mailto:eyang@yahoo-inc.com> > wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com
>>>>> <mailto:billgraham@gmail.com> > wrote:
>>>>> 
>>>>>> Thanks for your quick reply Eric.
>>>>>> 
>>>>>> The TsProcessor does use buildGenericRecord and has been working
fine for
>>>>>> me
>>>>>> (at least I thought it was). I've mapped it to my dataType as you
>>>>>> described
>>>>>> without problems. My only point with issue #1 was just that the
>>>>>> documentation
>>>>>> is off and that the DefaultProcessor yields what I think is unexpected
>>>>>> behavior.
>>>>>> 
>>>>> 
>>>>> I will update the documentation to align with the code.  Thank you for
>>>>> finding this.
>>>>> 
>>>>>> Yes, annotations would be useful. Or what about just having an extensions
>>>>>> directory (maybe lib/ext/) or something similar where custom jars
could
>>>>>> be
>>>>>> placed that are to be submitted by demux M/R? Do you know where the
code
>>>>>> resides that handles adding the chukwa-core jar? I poked around bit
but
>>>>>> couldn't find it.
>>>>>> 
>>>>>> Finally, is there a JIRA for this issue that you know of? If not
I'll
>>>>>> create
>>>>>> one. This is going to become a pain point for us soon, so if we have
a
>>>>>> design
>>>>>> I might be able to contribute a patch.
>>>>> 
>>>>> The extensions could be added by adding the class to the class path of
the
>>>>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>>>>> chukwa-demux-conf.xml, then it should work automatically.  We probably
>>>>> should have a jira to document this.  Please go ahead and file one.
>>>>> 
>>>>> For your interest, the annotation jira is:
>>>>> 
>>>>> https://issues.apache.org/jira/browse/CHUKWA-371
>>>>> <https://issues.apache.org/jira/browse/CHUKWA-371>
>>>>> 
>>>>> Regards,
>>>>> Eric
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 


Mime
View raw message