incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <>
Subject Re: How to integrate syslog-ng with chukwa
Date Wed, 17 Feb 2010 19:14:33 GMT

What you propose would certainly work. However, the whole
agent-collector pipeline isn't really necessary if your data already
comes in biggish files.  You might try out the Backfilling loader
instead. (bin/  The code is in, if you
need to read it. The idea is that you point it at a file, and it
copies that file directly into HDFS, appropriately formatted for

One warning. The backfilling loader was developed to meet the needs of
a particular site. Since it's a sort of special-purpose thing, we
haven't yet written the documentation.  You might be the second person
to use it in production.


2010/2/16 Guillermo PĂ©rez <>:
> Probably people here already have experience integrating syslog-ng and
> chukwa, so I want to let you know what we are planning and discuss
> possible improvements.
> We don't want to deploy chukwa agents to all the monitored servers
> (little disk and no java there), and use instead syslog-ng that is
> sent msgs through UDP to a central syslog server. I have setup it so
> it creates files with
> /var/log/cluster/$HOST/$YEAR/$MONTH/$DAY/$FACILITY pattern, and I'm
> planning importing this to chukwa with an agent and an adaptor
> DirTailingAdaptor on /var/log/cluster. Each day we will clean old
> temporary files based on path, to avoid rotations and problems with
> the adaptor.
> Is there any better option for doing this? Perhaps directly pipe from
> syslog to the chukwa agent? But I'm concerned about what will happen
> if the pipe doesn't work, the agent is not ready...
> Ideas and suggestions are welcome.
> Thanks a lot in advance!

Ari Rabkin
UC Berkeley Computer Science Department

View raw message