incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Ho to deploying a custom processor to demux
Date Wed, 23 Dec 2009 01:46:15 GMT
I thought this is the current implementation.  The class file should be in
the same package name, but it is not required to be in the same jar file.
If it is not working, please file a jira.

Regards,
Eric


On 12/22/09 4:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:

>> The extensions could be added by adding the class to the class path of the
>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>> chukwa-demux-conf.xml, then it should work automatically. 
> 
> Just to clarify, are you saying this is how it currently works or how it could
> work in the future?
> 
> Currently it doesn't work this way, which is the point of my post. I put a jar
> in the lib/ directory of my data processor that contained my processor. I
> mapped it in chukwa-demux.xml  and bounced the data processor. I did a ps and
> saw the jar in the DemuxManager classpath, but I still got
> ClassNotFoundExceptions.
> 
> The only way I could get it to work was to do the following:
> - Move my class into
> org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> - Add my compiled class to the chukwa-core jar.
> 
> 
> On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
>> 
>> 
>> 
>> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
>> 
>>> Thanks for your quick reply Eric.
>>> 
>>> The TsProcessor does use buildGenericRecord and has been working fine for me
>>> (at least I thought it was). I've mapped it to my dataType as you described
>>> without problems. My only point with issue #1 was just that the
>>> documentation
>>> is off and that the DefaultProcessor yields what I think is unexpected
>>> behavior.
>>> 
>> 
>> I will update the documentation to align with the code.  Thank you for
>> finding this.
>> 
>>> Yes, annotations would be useful. Or what about just having an extensions
>>> directory (maybe lib/ext/) or something similar where custom jars could be
>>> placed that are to be submitted by demux M/R? Do you know where the code
>>> resides that handles adding the chukwa-core jar? I poked around bit but
>>> couldn't find it.
>>> 
>>> Finally, is there a JIRA for this issue that you know of? If not I'll create
>>> one. This is going to become a pain point for us soon, so if we have a
>>> design
>>> I might be able to contribute a patch.
>> 
>> The extensions could be added by adding the class to the class path of the
>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>> chukwa-demux-conf.xml, then it should work automatically.  We probably
>> should have a jira to document this.  Please go ahead and file one.
>> 
>> For your interest, the annotation jira is:
>> 
>> https://issues.apache.org/jira/browse/CHUKWA-371
>> 
>> Regards,
>> Eric
>> 
>> 
> 
> 


Mime
View raw message