chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IvyTang <ivytang0...@gmail.com>
Subject Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Date Fri, 12 Jul 2013 07:04:12 GMT
Hi,Eric
In order to applying to all multiple datatypes in Demux,init the Reducer class depends on
the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.

The ChukwaRecordKey needs to add a  field named dataType , showing this ChukwaRecord's datatype
. And also i need add an argument in 
parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter
reporter) 

An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.

Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String
recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)

Is that acceptable? This is the only way i can think of to make the custom reducer more configurable.

-- 
Best regards,

Ivy Tang

On 2013-7-10, at 下午12:01, Eric Yang <eric818@gmail.com> wrote:

> Yes that will help.  Please include test case as well.  Thanks Ivy.
> 
> regards,
> Eric
> 
> 
> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <ivytang0812@gmail.com> wrote:
> 
>> This issue has been reported
>> https://issues.apache.org/jira/browse/CHUKWA-581.
>> 
>> And i think i can submit a patch later .
>> 
>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
>> 
>> This can implement reducer outside chukwa.
>> 
>> --
>> Best regards,
>> 
>> Ivy Tang
>> 
>> On 2013-7-9, at 下午1:22, Eric Yang <eric818@gmail.com> wrote:
>> 
>>> The recommended change can improve usability to implement parsers outside
>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for this?
>>> Thanks
>>> 
>>> regards,
>>> Eric
>>> 
>>> 
>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <ivytang0812@gmail.com> wrote:
>>> 
>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>> since we config the parserClass in config file.
>>>> 
>>>> But the ReduceProcessorFactory inits  the reducer class using the
>>>> classname
>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>> reduceType.
>>>> So the reducer class must be put in package
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>> reduceType.
>>>> 
>>>> Why not we put the reducer  class name in config file either?
>>>> For exam, the {customdatatype_mapper} means mapper class
>>>> ,{customdatatype_reducer} means reducer class.
>>>> 
>>>> This can give chukka more flexibility.
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>> Ivy Tang
>>>> 
>>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message