chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: chukwa for processing log4j logs
Date Tue, 19 Jul 2011 06:10:47 GMT
The easiest method to get going is to use trunk version of Chukwa, and setup your log4j configuration
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

This will setup log4j to stream data to localhost:9096
Then setup init_adaptors with:

add SocketAdaptor DataType 9000 0

This will start streaming data to HDFS.  You will need to build your own indexing of your
log data on HDFS to provide search function.  Hadoop has a tutorial for running "grep", it
may be a starting point to build a search service.  Hope this helps.

On Jul 18, 2011, at 10:26 PM, Ariel Rabkin wrote:

> Yes, this is a reasonable use for Chukwa.  You'll have an adaptor per
> log file. Depending on your particular needs, you can either have a
> custom log4j appender to set up the Chukwa side or else explicitly
> configure adaptors on the log files you care about.
> You will have to build the search part yourself.  Chukwa designed to
> get stuff into HDFS; the rest is up to you. Of course, if you build
> something generally usable and contribute it to the project, I'm sure
> you'd get help with development and maintenance.
> --Ari
> On Mon, Jul 18, 2011 at 10:21 PM, sangeetha <> wrote:
>> Hi,
>> I wanted to aggregate log4j logs files from different systems and perform
>> search on the logs.
>> Is it possible in chukwa? How can I use chukwa for this?
>> Let me know the configuration details for doing the same.
>> Thanks,
>> Sangeetha
>> --
>> View this message in context:
>> Sent from the Chukwa - Users mailing list archive at
> -- 
> Ari Rabkin
> UC Berkeley Computer Science Department

View raw message