incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <>
Subject Re: How to integrate syslog-ng with chukwa
Date Thu, 18 Feb 2010 17:31:09 GMT
*nod*  Syslog itself is a UDP-based protocol, if I'm not mistaken, and
there's no reason the collector couldn't simply listen for incoming
packets, and then turn them into Chunks.

Is that a plausible way to do this?

2010/2/18 Guillermo Pérez <>:
> On Wed, Feb 17, 2010 at 20:14, Ariel Rabkin <> wrote:
>> Hmm.
>> What you propose would certainly work. However, the whole
>> agent-collector pipeline isn't really necessary if your data already
>> comes in biggish files.  You might try out the Backfilling loader
>> instead. (bin/  The code is in
>>, if you
>> need to read it. The idea is that you point it at a file, and it
>> copies that file directly into HDFS, appropriately formatted for
>> Chukwa.
>> One warning. The backfilling loader was developed to meet the needs of
>> a particular site. Since it's a sort of special-purpose thing, we
>> haven't yet written the documentation.  You might be the second person
>> to use it in production.
> For now I will start with chukwa agent, so I can have minutes of
> latency before analyzing data in hadoop.
> It will be nice if the collector could directly grab from syslog-ng at
> some point, by udp, tcp or a pipe. That will make things lot faster.
> --
> Guille -ℬḭṩḩø- <>
> :wq

Ari Rabkin
UC Berkeley Computer Science Department

View raw message