incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian <engr...@gmail.com>
Subject Re: Ho to deploying a custom processor to demux
Date Wed, 23 Dec 2009 13:36:13 GMT
Hi Eric,

This isn't an issue for me yet, but I can see it becoming one if we end up
using Chukwa (currently investigating it). What's the reason behind
requiring the package name to be org.apache.hadoop.chukwa.
extraction.demux.processor.mapper?

Regards,
Chris


On Tue, Dec 22, 2009 at 6:46 PM, Eric Yang <eyang@yahoo-inc.com> wrote:

> I thought this is the current implementation.  The class file should be in
> the same package name, but it is not required to be in the same jar file.
> If it is not working, please file a jira.
>
> Regards,
> Eric
>
>
> On 12/22/09 4:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
>
> >> The extensions could be added by adding the class to the class path of
> the
> >> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
> >> chukwa-demux-conf.xml, then it should work automatically.
> >
> > Just to clarify, are you saying this is how it currently works or how it
> could
> > work in the future?
> >
> > Currently it doesn't work this way, which is the point of my post. I put
> a jar
> > in the lib/ directory of my data processor that contained my processor. I
> > mapped it in chukwa-demux.xml  and bounced the data processor. I did a ps
> and
> > saw the jar in the DemuxManager classpath, but I still got
> > ClassNotFoundExceptions.
> >
> > The only way I could get it to work was to do the following:
> > - Move my class into
> > org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> > - Add my compiled class to the chukwa-core jar.
> >
> >
> > On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> >>
> >>
> >>
> >> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
> >>
> >>> Thanks for your quick reply Eric.
> >>>
> >>> The TsProcessor does use buildGenericRecord and has been working fine
> for me
> >>> (at least I thought it was). I've mapped it to my dataType as you
> described
> >>> without problems. My only point with issue #1 was just that the
> >>> documentation
> >>> is off and that the DefaultProcessor yields what I think is unexpected
> >>> behavior.
> >>>
> >>
> >> I will update the documentation to align with the code.  Thank you for
> >> finding this.
> >>
> >>> Yes, annotations would be useful. Or what about just having an
> extensions
> >>> directory (maybe lib/ext/) or something similar where custom jars could
> be
> >>> placed that are to be submitted by demux M/R? Do you know where the
> code
> >>> resides that handles adding the chukwa-core jar? I poked around bit but
> >>> couldn't find it.
> >>>
> >>> Finally, is there a JIRA for this issue that you know of? If not I'll
> create
> >>> one. This is going to become a pain point for us soon, so if we have a
> >>> design
> >>> I might be able to contribute a patch.
> >>
> >> The extensions could be added by adding the class to the class path of
> the
> >> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
> >> chukwa-demux-conf.xml, then it should work automatically.  We probably
> >> should have a jira to document this.  Please go ahead and file one.
> >>
> >> For your interest, the annotation jira is:
> >>
> >> https://issues.apache.org/jira/browse/CHUKWA-371
> >>
> >> Regards,
> >> Eric
> >>
> >>
> >
> >
>
>

Mime
View raw message