flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cochran, David" <david.coch...@bsee.gov>
Subject Re: Automatic log analysis and alert generation
Date Mon, 26 Aug 2013 20:47:15 GMT

FWIW, here I'm using flume to pull all the logs to a central server and
then using a package called logcheck to scan the logs for keywords/events
periodically.  A bit of perl scripting to select the logs I want and such,
I just have it run every 30 minutes as that is frequent enough for my
purposes.  I'm scanning about 100 log files (most pretty active) and
runtime is about 2 mins with my crude perl scripting.

There are likely other packages out there that do things in a similar
fashion, but the logcheck scripts have served me well for over 15 years, so
to say the least it's inner workings are familiar. Might be worth looking


On Mon, Aug 26, 2013 at 3:22 PM, Mark Nuttall-Smith <
marknuttallsmith@gmail.com> wrote:

> Hi, I posted this question on stackoverflow (
> http://stackoverflow.com/questions/18448218/automatic-log-analysis-and-alert-generation),
> but thought I might get a better response here, so am crossposting... hope
> it's ok!
>  I would like some design advice for a centralized logging project I am
> considering. I have a number of components producing logs on various
> servers. Apache Flume looks like the sensible choice for streaming to a
> central log server, most likely into an elasticsearch instance for querying
> and analysis.
> Here's my question: I would like to provide a scripting engine listening
> to the flow of log events arriving on the central server. Would it make
> sense to do that as an interceptor in Flume, or as a plugin to
> elasticsearch, or something else completely?
> Thanks,
> Mark

View raw message