incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: Ho to deploying a custom processor to demux
Date Wed, 23 Dec 2009 00:40:43 GMT
> The extensions could be added by adding the class to the class path of the
> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
> chukwa-demux-conf.xml, then it should work automatically.

Just to clarify, are you saying this is how it currently works or how it
could work in the future?

Currently it doesn't work this way, which is the point of my post. I put a
jar in the lib/ directory of my data processor that contained my processor.
I mapped it in chukwa-demux.xml  and bounced the data processor. I did a ps
and saw the jar in the DemuxManager classpath, but I still got
ClassNotFoundExceptions.

The only way I could get it to work was to do the following:
- Move my class into
org.apache.hadoop.chukwa.extraction.demux.processor.mapper
- Add my compiled class to the chukwa-core jar.


On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com> wrote:

>
>
>
> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
>
> > Thanks for your quick reply Eric.
> >
> > The TsProcessor does use buildGenericRecord and has been working fine for
> me
> > (at least I thought it was). I've mapped it to my dataType as you
> described
> > without problems. My only point with issue #1 was just that the
> documentation
> > is off and that the DefaultProcessor yields what I think is unexpected
> > behavior.
> >
>
> I will update the documentation to align with the code.  Thank you for
> finding this.
>
> > Yes, annotations would be useful. Or what about just having an extensions
> > directory (maybe lib/ext/) or something similar where custom jars could
> be
> > placed that are to be submitted by demux M/R? Do you know where the code
> > resides that handles adding the chukwa-core jar? I poked around bit but
> > couldn't find it.
> >
> > Finally, is there a JIRA for this issue that you know of? If not I'll
> create
> > one. This is going to become a pain point for us soon, so if we have a
> design
> > I might be able to contribute a patch.
>
> The extensions could be added by adding the class to the class path of the
> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
> chukwa-demux-conf.xml, then it should work automatically.  We probably
> should have a jira to document this.  Please go ahead and file one.
>
> For your interest, the annotation jira is:
>
> https://issues.apache.org/jira/browse/CHUKWA-371
>
> Regards,
> Eric
>
>
>

Mime
View raw message