Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=Gat5Jn9tdKIrxEtQUD1IPjeUnT8i+LRu5x5HqwknQDMmq+jj/kcby7zzy+AGzRh2
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 23 Dec 2009 10:24:53 -0800
Subject: Re: Ho to deploying a custom processor to demux
From: Eric Yang <eyang@yahoo-inc.com>
To: <chukwa-user@hadoop.apache.org>
Message-ID: <C757A075.5993%eyang@yahoo-inc.com>
Thread-Topic: Ho to deploying a custom processor to demux
Thread-Index: AcqD/ThrTvb4GyyFkkymt374nUlFrA==
In-Reply-To: <39898D0E-044D-4C1E-88B2-8DC59E53149F@gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

I think this was the jira that you were talking about Ari.

https://issues.apache.org/jira/browse/CHUKWA-292

Both mapper and reducer should be able to live in another name space.  The
limitation is in the current code.  CHUKWA-422 is the current plan to update
Demux for hadoop 0.20+ and we can make it more user friendly in it.

Regards,
Eric


On 12/23/09 6:43 AM, "asrabkin@gmail.com" <asrabkin@gmail.com> wrote:

> I would say that limitation is a bug. I remember opening a Jira for it, once.
> Shouldn't be too hard to fix...
> 
> sent from my iPhone; please excuse typos and brevity.
> 
> On Dec 23, 2009, at 8:36 AM, Christian <engrean@gmail.com> wrote:
> 
>> Hi Eric,
>> 
>> This isn't an issue for me yet, but I can see it becoming one if we end up
>> using Chukwa (currently investigating it). What's the reason behind requiring
>> the package name to be org.apache.hadoop.chukwa.
>> extraction.demux.processor.mapper?
>> 
>> Regards,
>> Chris
>> 
>> 
>> On Tue, Dec 22, 2009 at 6:46 PM, Eric Yang <eyang@yahoo-inc.com
>> <mailto:eyang@yahoo-inc.com> > wrote:
>>> I thought this is the current implementation.  The class file should be in
>>> the same package name, but it is not required to be in the same jar file.
>>> If it is not working, please file a jira.
>>> 
>>> Regards,
>>> Eric
>>> 
>>> 
>>> On 12/22/09 4:40 PM, "Bill Graham" <billgraham@gmail.com
>>> <mailto:billgraham@gmail.com> > wrote:
>>> 
>>>>> The extensions could be added by adding the class to the class path of the
>>>>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>>>>> chukwa-demux-conf.xml, then it should work automatically.
>>>> 
>>>> Just to clarify, are you saying this is how it currently works or how it
>>>> could
>>>> work in the future?
>>>> 
>>>> Currently it doesn't work this way, which is the point of my post. I put a
>>>> jar
>>>> in the lib/ directory of my data processor that contained my processor. I
>>>> mapped it in chukwa-demux.xml  and bounced the data processor. I did a ps
>>>> and
>>>> saw the jar in the DemuxManager classpath, but I still got
>>>> ClassNotFoundExceptions.
>>>> 
>>>> The only way I could get it to work was to do the following:
>>>> - Move my class into
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper
>>>> - Add my compiled class to the chukwa-core jar.
>>>> 
>>>> 
>>>> On Tue, Dec 22, 2009 at 3:11 PM, Eric Yang <eyang@yahoo-inc.com
>>>> <mailto:eyang@yahoo-inc.com> > wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 12/22/09 2:40 PM, "Bill Graham" <billgraham@gmail.com
>>>>> <mailto:billgraham@gmail.com> > wrote:
>>>>> 
>>>>>> Thanks for your quick reply Eric.
>>>>>> 
>>>>>> The TsProcessor does use buildGenericRecord and has been working fine for
>>>>>> me
>>>>>> (at least I thought it was). I've mapped it to my dataType as you
>>>>>> described
>>>>>> without problems. My only point with issue #1 was just that the
>>>>>> documentation
>>>>>> is off and that the DefaultProcessor yields what I think is unexpected
>>>>>> behavior.
>>>>>> 
>>>>> 
>>>>> I will update the documentation to align with the code.  Thank you for
>>>>> finding this.
>>>>> 
>>>>>> Yes, annotations would be useful. Or what about just having an extensions
>>>>>> directory (maybe lib/ext/) or something similar where custom jars could
>>>>>> be
>>>>>> placed that are to be submitted by demux M/R? Do you know where the code
>>>>>> resides that handles adding the chukwa-core jar? I poked around bit but
>>>>>> couldn't find it.
>>>>>> 
>>>>>> Finally, is there a JIRA for this issue that you know of? If not I'll
>>>>>> create
>>>>>> one. This is going to become a pain point for us soon, so if we have a
>>>>>> design
>>>>>> I might be able to contribute a patch.
>>>>> 
>>>>> The extensions could be added by adding the class to the class path of the
>>>>> demux process.  If you put your jar file in CHUKWA_HOME/lib and update
>>>>> chukwa-demux-conf.xml, then it should work automatically.  We probably
>>>>> should have a jira to document this.  Please go ahead and file one.
>>>>> 
>>>>> For your interest, the annotation jira is:
>>>>> 
>>>>> https://issues.apache.org/jira/browse/CHUKWA-371
>>>>> <https://issues.apache.org/jira/browse/CHUKWA-371>
>>>>> 
>>>>> Regards,
>>>>> Eric
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>