chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thushara Wijeratna <thu...@gmail.com>
Subject Re: Alerting framework - feature idea
Date Wed, 04 Nov 2009 00:27:13 GMT
yeah, that makes sense. i don't have a strong argument, except it
might be a tad bit easier to integrate alerting to the system.
swatch is pretty good, however, for custom processing, for each
pattern matched, a separate process needs to be run. if alerts are
rare, as is generally the case, that is not a big problem.
one reason i'm considering Chukwa instead of swatch is that it
centralizes the input logs at the collector - swatch AFAIK doesn't
perform any centralization of logs.

thanks,
thushara

On Tue, Nov 3, 2009 at 3:51 PM, Ariel Rabkin <asrabkin@gmail.com> wrote:
> What you describe is certainly doable. I'm not sure what the use case
> is, though.
>
> The core goal for Chukwa is to facilitate MapReduce processing of
> logs. The idea of the SocketTeeWriter is to get a "sneak peek" at
> data, before it gets stored to HDFS. If collectors crash or get
> overloaded, data can get processed more than once by collectors. So
> there's a real cost to the real-time path.
>
> One of the main benefits of SocketTee is that the processing can
> happen in a separate process, or even on a separate machine.
> Integrating the pattern-matching in the pipeline is certainly doable,
> but it's not clear to me that that's an architecture we want to
> encourage or commit to.
>
> If people want Swatch, they know where to find it. What's the argument
> for needing to emulate it, real-time, in Chukwa?
>
> --Ari
>
> On Tue, Nov 3, 2009 at 3:48 PM, Thushara Wijeratna <thushw@gmail.com> wrote:
>> Would it be useful to provide something similar to the Swatch Log
>> monitoring for Chukwa?
>> http://www.linuxjournal.com/article/4776
>>
>> Currently, we can listen to port 9094 (after running a
>> SocketTeeWriter), and handle each input line.
>> I'm wondering whether there will be a value add in creating some more
>> infra-structure code in Chukwa that will:
>>
>> 1. do some regular expression parsing and filter the lines with the
>> alert condition(s)
>> 2. perform some standard actions, like email etc
>> 3. provide an interface to perform custom handling for the user
>>
>> The basic core will be someting like this:
>>
>> Interface AlertCallback {
>>
>>    boolean handle(String alertExp, String line);
>>
>> }
>>
>> Class AlertWriter extends PipelinableWriter {
>>    private String[] alertExps;
>>    private AlertCallback alertCB;
>>
>>    public AlertWriter(String[] alertExps, AlertCallback alertCB);
>> }
>>
>> It seems like most of the plumbing is already there, exposed in
>> SocketTeeWriter class, for ex: Filter class.
>> If you all think it is a good idea, I can help with this.
>>
>> thanks,
>> thushara
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Mime
View raw message