lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: Autocommit and replication have been slowing down
Date Tue, 23 Apr 2013 18:44:31 GMT
On 4/23/2013 11:27 AM, gustavonasu wrote:
>      We migrated recently from Solr 1.4 to 3.6.1. In the new version we have
> noticed that after some hours (around 8) the autocommit is taking more time
> to be executed.
>
>      In the new version we have noticed that after some hours the autocommit
> is taking more time to be executed. We configured autocommit with maxDocs=50
> and maxTime=10000ms but we've gotten few (3-5) minutes to index documents (I
> got this time seeing the docsPending on the Update Stats and refresh page.
> Is there another way to verify that information?).

Your question is a bit jumbled so I don't know exactly what you are 
saying for all of this, but I'll attempt to answer what I can.  Usually 
if your commits are taking a really long time, it means you're running 
into one of two problems:

1) It is taking a really long time to autowarm your Solr caches.  In 
most cases, it is the filterCache that takes the time, but not always. 
You can see how long it takes to warm the entire searcher as well as 
each individual cache in the Statistics page of the admin UI.  To fix 
this, you have to reduce the autowarmCount on your caches, reduce the 
complexity of your queries and filters or both.

2) Your Java heap is getting exhausted and Java is spending too much 
time doing full garbage collections so it can keep working.  Eventually 
this problem will result in OOM (Out of Memory) errors in your Solr log. 
  To fix this, raise your max heap, which is the -Xmx java option when 
starting your servlet container.  Raising the java heap might also 
require that you add physical RAM to your server.

On version 3.6, I believe that an index update/commit that results in 
segment merging will wait for that merging to complete.  If you do a lot 
of indexing, eventually you will run into a very large merge, and that 
can take a lot of time.  This would not explain why every autoCommit is 
taking a long time, though - it would only explain one out of dozens or 
hundreds.

>      A similar problem has been happening with the replication. We configured
> the pollInterval with 60s but the replication takes some minutes to be
> executed. You could see the timeElapsed value (around 6 minutes) on the
> Replication Stats.

If you optimize your index, or do enough index updates so that a large 
merge takes place, then a very large portion of your index will be 
comprised of brand new files, and if your index is large, that can take 
a long time to replicate.  It is also possible for the java heap problem 
(mentioned above) to cause this.

Thanks,
Shawn


Mime
View raw message