lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Huin <matthieu.h...@wallix.com>
Subject Re: lucene-based log searcher?
Date Fri, 14 Jan 2011 09:39:17 GMT
Hello,

I've been working on logs and metadata and I've been working on this 
specific problem for quite a while.

if you have a constant, predefined number of fields associated to each 
log, a relational database might perform just fine and offer you more 
possibilities regarding log analysis (statistics, aggregation, min/max 
... ). As a side note, you might not be aware that PostgreSQL ( > 8.3 ) 
comes with built-in full text search capabilities, so you don't have to 
sacrifice the functionality. A column-based database would probably be 
also worth trying.

On the other hand, if fields can vary in type and number for each log, 
the relational database model will perform very badly : if you normalize 
a lot you'll end up with a compact database schema that takes a lot of 
time to query ( and queries can become hellishly hard to write); if you 
don't you're losing most of what a database would bring you in the first 
place. This is a very close problem to the one discussed here : 
http://www.pui.ch/phred/archives/2005/04/tags-database-schemas.html this 
link is worth a read if you want to go on with databases.

Do not forget also that unless you use a column-based database, you 
might probably want to use indices on your tables in order to speed up 
some lookups ... But it will come at a hefty price at insertion time 
after you reach a certain amount of indexed logs.

Concerning my project (where we fall in the second case), current tests 
with lucene are very encouraging. Sure, we lose some analytics 
capabilities and the hard disk footprint is rather huge, but the query 
response time, be it from full-text searches or complex queries on 
specific fields, is outstanding.


Of course, in the end, it all depends on what you need to do. Just my 
personal experience here ...

Regards,

Matthieu Huin

Le 14/01/2011 06:47, chensheng a écrit :
> Hi, I have a similar case.
> If I only index some fixed fields with single-word/number, instead of full-text search,
> is Lucene any better than a relational database?
>
>
> ------------------ Original ------------------
> From:  "Benzion G"<benzionk@yahoo.com>;
> Date:  Fri, Jan 14, 2011 01:15 AM
> To:  "java-user"<java-user@lucene.apache.org>;
>
> Subject:  Re: lucene-based log searcher?
>
>
>
> Hi,
>
> I almost finished these days to write such a tool.
>
> It reads via FTPClient the log files and adds to Lucene index.
>
> Via small Web application you can see/filter/sort the log messages.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message