lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Recommended index-size per core
Date Fri, 12 May 2017 00:24:07 GMT
On 5/11/2017 4:59 PM, S G wrote:
> How can 50GB index be handled by a 10GB heap?
> I am a developer myself and would love to know as many details as possible.  So a long
answer would be much appreciated.

Lucene (which is what provides large pieces of Solr's functionality)
does not read the entire index into heap memory.  It only accesses the
parts of the index that it needs for the current query, and builds
certain structures in memory that it needs in order to process that
query.  Much of that gets thrown away as soon as the query is done, but
both Lucene and Solr do keep some of it in caches.

The precise details of what Lucene accesses and what memory structures
it uses are not known to me.  If you really want to know, the full
source code is available.

I have production servers running Solr that have well over 200GB of
index data and are running with a 13GB heap.  It is likely that I could
reduce that heap and still have no problems.

If there is free memory available, then large parts of your index will
be loaded into the operating system's OS disk cache and will remain
there, making Lucene fast.  Having enough spare memory for this is
essential for good performance with Lucene-based software like Solr.

Here's some more reading.  Disclaimer: I wrote the wiki page on the
second link to make supporting Solr on this mailing list easier.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
https://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn


Mime
View raw message