lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@mailarchiva.com>
Subject Re: search performance
Date Mon, 02 Jun 2014 11:50:08 GMT
Jack

First off, thanks for applying your mind to our performance problem.

On 2014/06/02, 1:34 PM, Jack Krupansky wrote:
> Do you have enough system memory to fit the entire index in OS system 
> memory so that the OS can fully cache it instead of thrashing with 
> I/O? Do you see a lot of I/O or are the queries compute-bound?
Nice idea. The index is 200GB, the machine currently has 128GB RAM. We 
are using SSDs, but disappointingly, installing them didn't reduce 
search times to acceptable levels. I'll have to check your last question 
regarding I/O... I assume it is I/O bound, though will double check...

Currently, we are using

fsDirectory = new NRTCachingDirectory(fsDir, 5.0, 60.0);

Are you proposing we increase maxCachedMB or use the RAMDirectory? With 
the latter, we will still need to persistent the index data to disk, as 
it is undergoing constant updates.
>
> You said you have a 128GB machine, so that sounds small for your 
> index. Have you tried a 256GB machine?
Nope..didn't think it would make much of a different. I suppose, 
assuming we could store the entire index in RAM it would be helpful. How 
does one do this with Lucene, while still persisting the data?
>
> How frequent are your commits for updates while doing queries?
Around ten to fifteen documents are being constantly added per second.

Thank again

Jamie


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message