lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Purdy <rdpu...@gmail.com>
Subject Major CPU performance problems under heavy user load with solr 1.2
Date Mon, 01 Oct 2007 15:42:34 GMT

Hi there, I am having some major CPU performance problems with heavy user
load with solr 1.2. I currently have approximately 4 million documents in
the index and I am doing some pretty heavy faceting on multi-valued columns.
I know that doing facets are expensive on multi-valued columns but the CPU
seems to max out (400%) with apache bench with just 5 identical concurrent
requests and I have the potential for a lot more concurrent requests then
that with my large number of users that hit our site per day and I am
wondering if there are any workarounds. Currently I am running the out of
the box solr solution (Example jetty application with my own schema.xml and
solrconfig.xml) on a dual Intel Duo core 64 bit box with 8 gigs of ram
allocated to the start.jar process dedicated to solr with no slaves. 

I have set up some aggressive caching in the solrconfig.xml for the
filtercache (class="solr.LRUCache"size="3000000" initialSize="2000000") and
have the HashDocSet set to 10000 to help with faceting, but still I am
getting some pretty poor performance. I have also tried autowarming the
facets by performing a query that hits all my multivalued facets with no
facet limits across all the documents in the index. This does seem to reduce
my query times by a lot because the filtercache grows to about 2.1 million
lookups and finishes the query in about 70 secs. However I have noticed an
issue with this because each time I do an optimize or a commit after
prewarming the facets the cache gets cleared, according to the stats on the
admin page, but the RSize does not shink for the process, and the queries
get slow again, so I prewarm the facets again and the memory usage keeps
growing like the cache is not being recycled and as a results the prewarm
query starts to get slower and slower as each time this occurs (after about
5 times of prewarms and then commit the query takes about 30 mins... ugh)
and almost run out of memory. 

Any thoughts on how to help improve this and fix the memory issue?
-- 
View this message in context: http://www.nabble.com/Major-CPU-performance-problems-under-heavy-user-load-with-solr-1.2-tf4549093.html#a12981540
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message