lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Poulton <Jon.Poul...@vyre.com>
Subject SolrCore has a large number of SolrIndexSearchers retained in "infoRegistry"
Date Wed, 23 Dec 2009 15:15:04 GMT
Hi there,
I'm looking at some problems we are having with some legacy code which uses Solr (1.3) under
the hood. We seem to get repeated OutOfMemory errors on a 24-48 hour basis searching a relatively
small index of 70,000 documents. This may be an error in the way Solr is configured, or it
may be a problem with the way it's being invoked, I don't know, as I'm fairly new to Solr.

The OutOfMemoryError is an unusual one:

java.lang.OutOfMemoryError: GC overhead limit exceeded

There are also a few stack traces in the solr logs. The first is:

 SEVERE: Error during auto-warming of key:root_show_stop_range:[2009-12-22T08:59:37.254 TO
*]:java.lang.OutOfMemoryError: GC overhead limit exceeded
                at java.util.Arrays.copyOfRange(Arrays.java:3209)
                at java.lang.String.<init>(String.java:215)
                at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
                at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:251)
                at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218)
                at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:55)
                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(MultiSegmentReader.java:609)
                at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next(MultiSegmentReader.java:560)
                at org.apache.lucene.search.RangeFilter.getDocIdSet(RangeFilter.java:268)
                at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116)
                at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81)
                at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
                at org.apache.lucene.search.Searcher.search(Searcher.java:126)
                at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:601)
                at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:507)
                at org.apache.solr.search.SolrIndexSearcher.cacheDocSet(SolrIndexSearcher.java:482)
                at org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:224)
                at org.apache.solr.search.LRUCache.warm(LRUCache.java:194)
                at org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1518)
                at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)

I took a memory dump of the running application, and found that vast amounts of memory (over
400MB) was being consumed by nine or ten large SolrIndexSearcher objects; references to which
are being held within a LinkedHashMap in SolrCore called "infoRegistry".

I had a quick look at the Solr 1.3.0 source code to try and figure out what was going wrong
and whether SolrCore was being used incorrectly in our own source. It looks like whenever
a new Searcher is created, it "registers" itself with SolrCore, and this registration places
a reference to the Searcher in a LinkedHashMap (the "infoRegistry").

What is puzzling me is why so many SolrIndexSearcher objects are being created, and what the
conditions are for their creation and removal. The code I can see in our own product does
not use SolrIndexSearcher directly, it simply makes calls to "execute" on SolrCore; so I would
normally expect that that class would be managing the Searcher life cycle internally.

Does anyone have any idea as to what may be going on here? I have had a look at the solrconfig.xml
file, but it does not seem to depart significantly from the defaults provided.

Thanks in advance for any help.

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message