chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IvyTang <ivytang0...@gmail.com>
Subject Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Date Fri, 12 Jul 2013 09:01:45 GMT
Fix:
  Don't need to add the argument dataType in AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey,
ChukwaRecord> output, Reporter reporter).
  I can get the datatype  from the chunk.
-- 
Best regards,

Ivy Tang

On 2013-7-12, at 下午3:04, IvyTang <ivytang0812@gmail.com> wrote:

> Hi,Eric
> In order to applying to all multiple datatypes in Demux,init the Reducer class depends
on the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.
> 
> The ChukwaRecordKey needs to add a  field named dataType , showing this ChukwaRecord's
datatype . And also i need add an argument in 
> parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output,
Reporter reporter) 
> 
> An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.
> 
> Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String
recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
> 
> Is that acceptable? This is the only way i can think of to make the custom reducer more
configurable.
> 
> -- 
> Best regards,
> 
> Ivy Tang
> 
> On 2013-7-10, at 下午12:01, Eric Yang <eric818@gmail.com> wrote:
> 
>> Yes that will help.  Please include test case as well.  Thanks Ivy.
>> 
>> regards,
>> Eric
>> 
>> 
>> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <ivytang0812@gmail.com> wrote:
>> 
>>> This issue has been reported
>>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>> 
>>> And i think i can submit a patch later .
>>> 
>>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
>>> 
>>> This can implement reducer outside chukwa.
>>> 
>>> --
>>> Best regards,
>>> 
>>> Ivy Tang
>>> 
>>> On 2013-7-9, at 下午1:22, Eric Yang <eric818@gmail.com> wrote:
>>> 
>>>> The recommended change can improve usability to implement parsers outside
>>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for this?
>>>> Thanks
>>>> 
>>>> regards,
>>>> Eric
>>>> 
>>>> 
>>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <ivytang0812@gmail.com> wrote:
>>>> 
>>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>>> since we config the parserClass in config file.
>>>>> 
>>>>> But the ReduceProcessorFactory inits  the reducer class using the
>>>>> classname
>>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>>> reduceType.
>>>>> So the reducer class must be put in package
>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>>> reduceType.
>>>>> 
>>>>> Why not we put the reducer  class name in config file either?
>>>>> For exam, the {customdatatype_mapper} means mapper class
>>>>> ,{customdatatype_reducer} means reducer class.
>>>>> 
>>>>> This can give chukka more flexibility.
>>>>> 
>>>>> --
>>>>> Best regards,
>>>>> 
>>>>> Ivy Tang
>>>>> 
>>>>> 
>>> 
>>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message