flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wolfgang Hoschek <whosc...@cloudera.com>
Subject Re: Automatic log analysis and alert generation
Date Mon, 26 Aug 2013 21:12:04 GMT
Take a look at the Apache Flume Morphline Solr Sink, for example for heavy duty ETL processing
and
ingestion into Solr:

       http://flume.apache.org/FlumeUserGuide.html#morphlinesolrsink

It provides a scripting engine that enables CEP on the flow of log events.

Wolfgang.

On Aug 26, 2013, at 1:22 PM, Mark Nuttall-Smith wrote:

> Hi, I posted this question on stackoverflow (http://stackoverflow.com/questions/18448218/automatic-log-analysis-and-alert-generation),
but thought I might get a better response here, so am crossposting... hope it's ok!
> 
> I would like some design advice for a centralized logging project I am considering. I
have a number of components producing logs on various servers. Apache Flume looks like the
sensible choice for streaming to a central log server, most likely into an elasticsearch instance
for querying and analysis.
> 
> Here's my question: I would like to provide a scripting engine listening to the flow
of log events arriving on the central server. Would it make sense to do that as an interceptor
in Flume, or as a plugin to elasticsearch, or something else completely?
> 
> Thanks,
> 
> Mark
> 
> 


Mime
View raw message