lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: solr OOM Crash
Date Thu, 12 Dec 2013 21:08:36 GMT
Hi Sandra,

Not a direct answer, but if you are seeing this around merges, have you
tried relaxing the merge factor to, say, 10?

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Dec 12, 2013 at 12:10 PM, Sandra Scott <scottsandra85@gmail.com>wrote:

> Helllo,
>
> We are experiencing unexplained OOM crashes. We have already seen it a few
> times, over our different solr instances. The crash happens only at a
> single shard of the collection.
>
> Environment details:
> 1. Solr 4.3, running on tomcat.
> 2. 24 Shards.
> 3. Indexing rate of ~800 docs per minute.
>
> Solrconfig.xml:
> 1. Merge factor 4
> 2. Sofrcommit every 10 min
> 3. Hardcommit every 30 min
>
> Main findings:
> 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount of
> soft and hard commits in comparison to other shards.
> 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb
> resourced to the JVM, mainly referenced by CompressingStoredFieldsReader GC
> root (which by looking at the code, we suspect they were created due to
> CompressingSortedFieldsWriter.merge).
>
> Sub findings:
> 1. GC logs: Showed 108 GC fails prior to the crash.
> 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays
> high 6 min before the OOM.
> 3. Memory: Half an hour before OOM the usage slowly rises, until it gets to
> 5.4gb.
>
> Has anyone encountered higher than normal commit rate that seem to increase
> merge rate and cause what I described?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message