lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Disable or limit the size of Lucene field cache
Date Thu, 09 Apr 2015 23:30:03 GMT
On 4/9/2015 4:36 PM, pras.venkatesh wrote:
> I have a solr deployment (version 4.8.1) and I often encounter memory leak
> followed by JVM crash, have around 450 million documents in the collection. 
> The schema has 42 fields and clients can sort on the fields, not doing any
> faceting.
>
> Now every time I collect the Heap Dump, I see lucene Field Cache consuming
> the entire heap and causing the JVM to crash. I am looking for ways to 
>
> a. set an upper bound for lucene field cache
> b. how to use field cache effectively so that it doesn't grow.
> c. Is there a way to turn off the field cache

Chances are that it's not an actual memory leak.  Memory leaks *do*
sometimes happen in Solr and Lucene, but they have been quite rare.  I
would imagine that the nature of your index config and your queries
simply requires a larger amount of heap memory than you have configured.

You cannot set an upper bound for the FieldCache.  It will use as much
memory as it needs to complete the operation that has been requested. 
To get rid of the FieldCache entirely, you must upgrade to Solr 5.0 and
use docValues for sorting and facets.

If many of your fields are used for sorting on 450 million documents,
that will result in a large amount of memory being used in the FieldCache.

If you enable docValues for any fields that will be used for sorting,
then completely reindex, your memory usage should be dramatically
lower.  If you enable docValues but don't reindex, sorting will not
work.  I do notknow if you will get an error, but it will not function.

https://cwiki.apache.org/confluence/display/solr/DocValues
http://wiki.apache.org/solr/HowToReindex

There is a wiki page that describes some of the things that can cause
high heap usage, and some possible mitigation steps:

http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Thanks,
Shawn


Mime
View raw message