incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <Stuti_Awas...@persistent.co.in>
Subject RE: Changing default demuxer
Date Wed, 29 Sep 2010 12:43:36 GMT
Hi all,



I have applied the patch and set the property in chukwa-demux-conf.xml file as follows :

<property>

 <name>chukwa.demux.mapper.default.processor</name>

 <value>com.test.genericdemuxer.xyz</value>

</property>



<property>

 <name>chukwa.demux.reducer.default.processor</name>

 <value>com.test.genericdemuxer.xyz</value>

</property>

I tested the whole process from log collection to creation of .evt files in repos, that worked
fine.

Please note I have not added any datatype for my logtype in demux-conf file so that it can
go to my processor that I have set in the property file.



I also tested the whole process again without these entries in the demux-conf file to check
the default behavior of chukwa, but surprisingly that also worked fine without any error.



There was a difference in the sizes of the .evt file that generated from both the process.
I am not able to figure out that whether my processor is getting used or not.



How can I get to know that it is using my processor?



I also have a query that in demux.java , we have code where  DefaultProcessor is hardcoded,
and in documentation it is written that TSProcessor will the default processor will be used
if we do not specify the datatype for particular log type in demux-con.xml file.
String defaultProcessor = Demux.jobConf.get(
            "chukwa.demux.mapper.default.processor",
            "org.apache.hadoop.chukwa.extraction.demux.processor.mapper.DefaultProcessor");

String processorClass = Demux.jobConf.get(chunk.getDataType(),
                defaultProcessor);



I think that either I am missing something or not able to clearly understand the flow.

Please suggest.



Regards

Stuti Awasthi



-----Original Message-----

From: Bill Graham [mailto:billgraham@gmail.com]

Sent: Tuesday, September 28, 2010 9:32 AM

To: chukwa-user@incubator.apache.org

Cc: Jaydeep Ayachit

Subject: Re: Changing default demuxer



This can be changed with the following configs:



chukwa.demux.mapper.default.processor

chukwa.demux.reducer.default.processor



Reference JIRA:

https://issues.apache.org/jira/browse/CHUKWA-473



This is part from Chukwa 0.4.0.





On Mon, Sep 27, 2010 at 5:47 AM, Jaydeep Ayachit

<jaydeep_ayachit@persistent.co.in> wrote:

> Hello,

>

>

>

> Chukwa by default uses TSProcessor demuxer for data types that are not

> defined in chukwa-demux-config file.

>

> Can this be changed through some config file?

>

>

>

> I looked through the code and saw that TSProcessor is hard coded as default

> demuxer.

>

> I am using chukwa 0.4

>

>

>

> Regards

>

> Jaydeep

>

>

>

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.

Mime
View raw message