lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Solr 4.10.2 Cores in Recovery
Date Tue, 22 Sep 2015 23:42:52 GMT
On 9/22/2015 11:54 AM, vsilgalis wrote:
> I've actually read that article a few times.
>
> Yeah I know we aren't perfect in opening searchers. Yes we are committing
> from the client, this is something that is changing in our next code
> release, AND we are auto soft committing every second.  
>
> <filterCache class="solr.FastLRUCache" size="32768" initialSize="32768"
> autowarmCount="256"/>
> <queryResultCache class="solr.LRUCache" size="32768" initialSize="32768"
> autowarmCount="256"/>
> <documentCache class="solr.LRUCache" size="32768" initialSize="32768"
> autowarmCount="256"/>

Those are huge caches.  Especially the filterCache, because each filter
entry can be megabytes in size, depending on how many documents are in
the core.  If your index ever reaches the point where the filterCache
can grow to thousands of entries, your heap memory usage may grow out of
control.

The documentCache cannot autowarm, so that autowarmCount setting is
irrelevant.  The other two are important, and 256 is a pretty large
number for that setting.  It is unlikely that your autowarming completes
in less than one second.

I've repeated some of what Erick already told you, but I would like to
add the following.  On your autoSoftCommit interval of one second, the
article that Erick linked has this to say:

-------
Set your soft commit interval to as long as you can stand. Don't listen
to your product manager who says "we need no more than 1 second
latency". Really. Push back hard and see if the /user/ is best served or
will even notice. Soft commits and NRT are pretty amazing, but they’re
not free.
-------

This autoSoftCommit interval, especially with large indexes, can cause a
performance death spiral.  In SolrCloud, that death spiral tends to
cause constant replica recovery.  A previous message you sent to the
list indicated that your shards are each 10GB in size, which counts as a
large index.  Many people have indexes that are larger, but that's still
pretty big.

Thanks,
Shawn


Mime
View raw message