incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From asrab...@gmail.com
Subject Re: Ho to deploying a custom processor to demux
Date Wed, 23 Dec 2009 14:43:24 GMT
I would say that limitation is a bug. I remember opening a Jira for  
it, once. Shouldn't be too hard to fix...

sent from my iPhone; please excuse typos and brevity.

On Dec 23, 2009, at 8:36 AM, Christian <engrean@gmail.com> wrote:

> Hi Eric,
>
> This isn't an issue for me yet, but I can see it becoming one if we  
> end up using Chukwa (currently investigating it). What's the reason  
> behind requiring the package name to be org.apache.hadoop.chukwa.
> extraction.demux.processor.mapper?
>
> Regards,
> Chris
>
>
> On Tue, Dec 22, 2009 at 6:46 PM, Eric Yang <eyang@yahoo-inc.com>  
> wrote:
> I thought this is the current implementation.  The class file should  
> be in
> the same package name, but it is not required to be in the same jar  
> file.
> If it is not working, please file a jira.
>
> Regards,
> Eric
>
>
> On 12/22/09 4:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
>
> >> The extensions could be added by adding the class to the class  
> path of the
> >> demux process.  If you put your jar file in CHUKWA_HOME/lib and  
> update
> >> chukwa-demux-conf.xml, then it should work automatically.
> >
> > Just to clarify, are you saying this is how it currently works or  
> how it could
> > work in the future?
> >
> > Currently it doesn't work this way, which is the point of my post.  
> I put a jar
> > in the lib/ directory of my data processor that contained my  
> processor. I
> > mapped it in chukwa-demux.xml  and bounced the data processor. I  
> did a ps and
> > saw the jar in the DemuxManager classpath, but I still got
> > ClassNotFoundExceptions.
> >
> > The only way I could get it to work was to do the following:
> > - Move my class into
> > org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> > - Add my compiled class to the chukwa-core jar.
> >
> >
> > On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com>  
> wrote:
> >>
> >>
> >>
> >> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com> wrote:
> >>
> >>> Thanks for your quick reply Eric.
> >>>
> >>> The TsProcessor does use buildGenericRecord and has been working  
> fine for me
> >>> (at least I thought it was). I've mapped it to my dataType as  
> you described
> >>> without problems. My only point with issue #1 was just that the
> >>> documentation
> >>> is off and that the DefaultProcessor yields what I think is  
> unexpected
> >>> behavior.
> >>>
> >>
> >> I will update the documentation to align with the code.  Thank  
> you for
> >> finding this.
> >>
> >>> Yes, annotations would be useful. Or what about just having an  
> extensions
> >>> directory (maybe lib/ext/) or something similar where custom  
> jars could be
> >>> placed that are to be submitted by demux M/R? Do you know where  
> the code
> >>> resides that handles adding the chukwa-core jar? I poked around  
> bit but
> >>> couldn't find it.
> >>>
> >>> Finally, is there a JIRA for this issue that you know of? If not  
> I'll create
> >>> one. This is going to become a pain point for us soon, so if we  
> have a
> >>> design
> >>> I might be able to contribute a patch.
> >>
> >> The extensions could be added by adding the class to the class  
> path of the
> >> demux process.  If you put your jar file in CHUKWA_HOME/lib and  
> update
> >> chukwa-demux-conf.xml, then it should work automatically.  We  
> probably
> >> should have a jira to document this.  Please go ahead and file one.
> >>
> >> For your interest, the annotation jira is:
> >>
> >> https://issues.apache.org/jira/browse/CHUKWA-371
> >>
> >> Regards,
> >> Eric
> >>
> >>
> >
> >
>
>

Mime
View raw message