lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Latency in updates.
Date Fri, 12 Oct 2018 17:42:04 GMT
On 10/12/2018 8:46 AM, root23 wrote:
> We are having an issue where we are seeing latency in updates. We are on
> solr 6.
> The new documents are reflected right away but updates to existing document
> take sometime from 30 seconds to couple of minutes.
>
> This is some relevant things from our solrconfig.
>
> Our autosoft commit time is 500ms.(I know its low.)
> <autoSoftCommit>
> <maxTime>500</maxTime>
> </autoSoftCommit>

That interval is WAY too low.  The starting point I recommend for 
autoSoftCommit is 60000 -- one minute.  Depending on everything you're 
dealing with, you might want to go up or down from there.  The number 
you ultimately end up with should be based on two things:

* How long it actually takes to open and warm a new searcher.
* A REALISTIC assessment of how quickly users must be able to see index 
changes.

> <ramBufferSizeMB>2048</ramBufferSizeMB>

You're just throwing away memory here.  The default buffer size is 
100MB.  Some quick testing by Solr developers a long time ago indicated 
that values above 128MB do not improve performance. Unless your Solr 
documents are HUGE, you're not going to gain anything by making it 
bigger, except a need for a very large heap -- especially when 
allocating a buffer size measured in gigabytes.

> <int name="maxThreadCount">8</int>
> <int name="maxMergeCount">16</int>

For most situations, the merge scheduler doesn't need more than 1 for 
threads and 6 for merges.  Since you're on SSD, increasing threads won't 
cause problems, but in my opinion you shouldn't go higher than about 3 
or 4.  It is extremely unlikely that you'll ever have 16 merges 
scheduled simultaneously ... I've never seen more than about 4, and 
that's from indexing MILLIONS of documents.

> Also in our logs i see following .
> Lucene flush took 0.003 seconds with 2 segments flushed and 0.022 MB flushed
>
> SolrMetricsEventListener.java:65 - Lucene flush took 0.001 seconds with 1
> segments flushed and 0.009 MB flushed
>
> SolrMetricsEventListener.java:65 - Lucene flush took 0.006 seconds with 2
> segments flushed and 0.057 MB flushed
>
> SolrMetricsEventListener.java:65 - Lucene flush took 0.008 seconds with 5
> segments flushed and 0.046 MB flushed
>
> so if you see we are flushing too often and very small amount . May be its
> because of the low autosoft commit time ?

Yes, I think the small flushes would indeed be from your low 
autoSoftCommit time.  Small segments aren't necessarily a problem, in 
the end that means there will be a lot more segment merging, but with 
small segments, the lower level merges will be quick.  But it does mean 
that the same data can be merged more times than necessary.

> Can someone explain to me what is going on. Are these things related.
> And why is that new document addition is fast but updates take time. I know
> updates have to do a delete first. Is that a reason ?

I have no idea why there would be a discrepancy between those actions.  
At the Lucene level, the work done is pretty much identical either way.  
My best guess about why you would have ANY kind of latency issues is the 
amount of time needed to open a new searcher, as Erick was talking 
about.  Can you share your entire solrconfig.xml file?  Attachments 
rarely make it to the list -- use a paste website or a file sharing website.

Thanks,
Shawn


Mime
View raw message