lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Lucene performance
Date Sat, 25 Jan 2014 12:44:26 GMT
You'll have to do some tuning with that kind of ingestion rate, and
you're talking about a significant size cluster here. At 172M
documents/day or so, you're not going to store very many days per
node.

Storing doesn't make much of any difference as far as search
speed is concerned, the raw data is stored in separate files
(*.fdt and *.fdx files) and doesn't affect _search_. They are
accessed to assemble the response.

Otherwise there's not a lot of info to go on here. Here are some
resources:
http://wiki.apache.org/solr/SolrPerformanceFactors
http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Best,
Erick

On Sat, Jan 25, 2014 at 1:27 AM, Hamed Ghavamnia <ghavamnia.h@gmail.com> wrote:
> Hello,
>
> I searched a lot about lucene limits and its performance, but I still don't
> know how much I can count on it. I'm storing logs and indexing them with
> lucene. The event per second is 2000. The format of each log is generally
> 'fieldname' : 'fieldvalue'.
> What search performance should I expect after a few days. Right now I'm
> having around 25 seconds of query response time on around 500 million logs.
> Each log is converted into a document and the field values are stored as
> well as being indexed. I have around 10 fields in each log.
> Is my query time normal, of am I making a huge mistake?
> How much does storing fields make a difference, would it be better if I
> didn't store the fields.
>
> Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message