Mailing-List: contact chukwa-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of ivytang0812@gmail.com
 designates 209.85.192.182 as permitted sender)
From: IvyTang <ivytang0812@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_EFAAE08F-33CC-4C2F-8611-E8D05DAE9450"
Message-Id: <ECF7D1D6-1931-499D-A8A9-080E7B239C52@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: About CHUKWA-581,Demux do not support user custom reducer package
 name.
Date: Fri, 12 Jul 2013 15:04:12 +0800
References: <44798F96-3C3A-41A3-9D5B-8C540E8BC0BB@gmail.com>
 <CADfbTpFzOzwpyWAqVfxOzHJjXYhQyg9ynS4f2DD6+c0YYpb_qA@mail.gmail.com>
 <F92E6D60-FF62-4D42-B301-62CA63C61A1B@gmail.com>
 <CADfbTpEfeySGEJa=D0mmo_-BS2WNOdZa8Dy6ufMmW6xEaf+Xeg@mail.gmail.com>
To: chukwa-dev@incubator.apache.org
In-Reply-To: 
 <CADfbTpEfeySGEJa=D0mmo_-BS2WNOdZa8Dy6ufMmW6xEaf+Xeg@mail.gmail.com>

--Apple-Mail=_EFAAE08F-33CC-4C2F-8611-E8D05DAE9450
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=GB2312

Hi,Eric
In order to applying to all multiple datatypes in Demux,init the Reducer =
class depends on the ChukwaRecordKey's datatype . I need the datatype in =
ChukwaRecordKey.

The ChukwaRecordKey needs to add a  field named dataType , showing this =
ChukwaRecord's datatype . And also i need add an argument in=20
parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> =
output, Reporter reporter)=20

An argument String dataType needs to be passed to this method to build =
the ChukwaRecordKey.

Adding a field in ChukwaRecordKey ,and add an argument in method =
AbstractProcessor.parse(String recordEntry, =
OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter =
reporter)

Is that acceptable? This is the only way i can think of to make the =
custom reducer more configurable.

--=20
Best regards,

Ivy Tang

On 2013-7-10, at =CF=C2=CE=E712:01, Eric Yang <eric818@gmail.com> wrote:

> Yes that will help.  Please include test case as well.  Thanks Ivy.
>=20
> regards,
> Eric
>=20
>=20
> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <ivytang0812@gmail.com> =
wrote:
>=20
>> This issue has been reported
>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>=20
>> And i think i can submit a patch later .
>>=20
>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
>>=20
>> This can implement reducer outside chukwa.
>>=20
>> --
>> Best regards,
>>=20
>> Ivy Tang
>>=20
>> On 2013-7-9, at =CF=C2=CE=E71:22, Eric Yang <eric818@gmail.com> =
wrote:
>>=20
>>> The recommended change can improve usability to implement parsers =
outside
>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for =
this?
>>> Thanks
>>>=20
>>> regards,
>>> Eric
>>>=20
>>>=20
>>> On Mon, Jul 8, 2013 at 6:54 AM, =CC=C6 =D3=B1 =
<ivytang0812@gmail.com> wrote:
>>>=20
>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>> since we config the parserClass in config file.
>>>>=20
>>>> But the ReduceProcessorFactory inits  the reducer class using the
>>>> classname
>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>> reduceType.
>>>> So the reducer class must be put in package
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and =
named
>>>> reduceType.
>>>>=20
>>>> Why not we put the reducer  class name in config file either?
>>>> For exam, the {customdatatype_mapper} means mapper class
>>>> ,{customdatatype_reducer} means reducer class.
>>>>=20
>>>> This can give chukka more flexibility.
>>>>=20
>>>> --
>>>> Best regards,
>>>>=20
>>>> Ivy Tang
>>>>=20
>>>>=20
>>=20
>>=20


--Apple-Mail=_EFAAE08F-33CC-4C2F-8611-E8D05DAE9450--