lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Long blocking during indexing + deleteByQuery
Date Tue, 07 Nov 2017 16:51:39 GMT
On 11/5/2017 12:20 PM, Chris Troullis wrote:
> The issue I am seeing is when some
> threads are adding/updating documents while other threads are issuing
> deletes (using deleteByQuery), solr seems to get into a state of extreme
> blocking on the replica

The deleteByQuery operation cannot coexist very well with other indexing
operations.  Let me tell you about something I discovered.  I think your
problem is very similar.

Solr 4.0 and later is supposed to be able to handle indexing operations
at the same time that the index is being optimized (in Lucene,
forceMerge).  I have some indexes that take about two hours to optimize,
so having indexing stop while that happens is a less than ideal
situation.  Ongoing indexing is similar in many ways to a merge, enough
that it is handled by the same Merge Scheduler that handles an optimize.

I could indeed add documents to the index without issues at the same
time as an optimize, but when I would try my full indexing cycle while
an optimize was underway, I found that all operations stopped until the
optimize finished.

Ultimately what was determined (I think it was Yonik that figured it
out) was that *most* indexing operations can happen during the optimize,
*except* for deleteByQuery.  The deleteById operation works just fine.

I do not understand the low-level reasons for this, but apparently it's
not something that can be easily fixed.

A workaround is to send the query you plan to use with deleteByQuery as
a standard query with a limited fl parameter, to retrieve matching
uniqueKey values from the index, then do a deleteById with that list of
ID values instead.

Thanks,
Shawn


Mime
View raw message