chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: chukwa for processing log4j logs
Date Tue, 19 Jul 2011 06:10:47 GMT
The easiest method to get going is to use trunk version of Chukwa, and setup your log4j configuration
like:

log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
log4j.appender.DRFA.RemoteHost=localhost
log4j.appender.DRFA.Port=9000
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

This will setup log4j to stream data to localhost:9096
Then setup init_adaptors with:

add SocketAdaptor DataType 9000 0

This will start streaming data to HDFS.  You will need to build your own indexing of your
log data on HDFS to provide search function.  Hadoop has a tutorial for running "grep", it
may be a starting point to build a search service.  Hope this helps.

regards,
Eric
On Jul 18, 2011, at 10:26 PM, Ariel Rabkin wrote:

> Yes, this is a reasonable use for Chukwa.  You'll have an adaptor per
> log file. Depending on your particular needs, you can either have a
> custom log4j appender to set up the Chukwa side or else explicitly
> configure adaptors on the log files you care about.
> 
> You will have to build the search part yourself.  Chukwa designed to
> get stuff into HDFS; the rest is up to you. Of course, if you build
> something generally usable and contribute it to the project, I'm sure
> you'd get help with development and maintenance.
> 
> --Ari
> 
> On Mon, Jul 18, 2011 at 10:21 PM, sangeetha <get2sangi@yahoo.com> wrote:
>> Hi,
>> 
>> I wanted to aggregate log4j logs files from different systems and perform
>> search on the logs.
>> Is it possible in chukwa? How can I use chukwa for this?
>> 
>> Let me know the configuration details for doing the same.
>> 
>> Thanks,
>> Sangeetha
>> 
>> --
>> View this message in context: http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3181483.html
>> Sent from the Chukwa - Users mailing list archive at Nabble.com.
>> 
> 
> 
> 
> -- 
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department


Mime
View raw message