lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Smith <>
Subject Re: Searching Log Files
Date Tue, 14 Oct 2008 21:34:11 GMT

On 15/10/2008, at 7:37 AM, Chris Gilliam wrote:

> Hello Everyone,
> New to Lucene..
> We currently roughly 100Gig of log files.  We are needing to build  
> a search
> application that can return rows of data from the files and combine  
> the
> results?
> Does Lucene index the content in the files?
> Will it be able to find  matching criteria say a date and then  
> return the
> next X number of lines?
> The indexing piece, how does it work if I am looking for something  
> that has
> been added to the file since the indexing has taken place?

I'm working on just such a thing over on my Apache Lab project called  

It is designed to process log4j environments, soaking from network- 
based appenders, of by importing log files based on a pattern.  It's  
relatively early days but the pinpoint-service layer recently ran  
through a production load test environment soaking 500-1000 events  
per/second no problems at all.

The pinpoint-search layer provides an API to query a  
'context' (usually a days worth of logs) to find matching events, and  
then to be able to view the temporal context of specific matching  
events so you can see them inline with their related logs (either by  
the Thread that created the original matching event, +/-10 minutes,  
or by showing all logging events around that time +/- 10 minutes).

Really good for production triage.  I'm trying to work on a Web-based  
app that allows a dynamic sort-of-OLAP analysis of log files (charts,  
trends, etc).



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message