incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: How to integrate syslog-ng with chukwa
Date Thu, 18 Feb 2010 17:31:09 GMT
*nod*  Syslog itself is a UDP-based protocol, if I'm not mistaken, and
there's no reason the collector couldn't simply listen for incoming
packets, and then turn them into Chunks.

Is that a plausible way to do this?

2010/2/18 Guillermo Pérez <bisho@tuenti.com>:
> On Wed, Feb 17, 2010 at 20:14, Ariel Rabkin <asrabkin@gmail.com> wrote:
>> Hmm.
>>
>> What you propose would certainly work. However, the whole
>> agent-collector pipeline isn't really necessary if your data already
>> comes in biggish files.  You might try out the Backfilling loader
>> instead. (bin/backfill.sh)  The code is in
>> org.apache.hadoop.chukwa.tools.backfilling.BackfillingLoader, if you
>> need to read it. The idea is that you point it at a file, and it
>> copies that file directly into HDFS, appropriately formatted for
>> Chukwa.
>>
>> One warning. The backfilling loader was developed to meet the needs of
>> a particular site. Since it's a sort of special-purpose thing, we
>> haven't yet written the documentation.  You might be the second person
>> to use it in production.
>
> For now I will start with chukwa agent, so I can have minutes of
> latency before analyzing data in hadoop.
>
> It will be nice if the collector could directly grab from syslog-ng at
> some point, by udp, tcp or a pipe. That will make things lot faster.
>
> --
> Guille -ℬḭṩḩø- <bisho@tuenti.com>
> :wq
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message