lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: [Migration Solr5 to Solr6] Unwanted deleted files references
Date Tue, 14 Mar 2017 16:42:15 GMT
Yeah, it's a little confusing. But SolrQueryReqeustBase.getSearcher
calls, in turn, core.getSearcher which explicitly says in the
javadocs:

* If returnSearcher==true</tt> then a SolrIndexSearcher will be returned with
* the reference count incremented.  It must be decremented when no
longer needed.

See a similar pattern in IndexFetcher:

  searcher = core.getSearcher(true, true, waitSearcher, true);
try {
   blah blah blah
} finally {
  if (searcher != null) {
    searcher.decref();
  }
  core.close();
}

This is so fundamental to Solr operating at _all_ that I'd lay long
odds this is just confusing, not a bug or everybody would be running
out of file handles.

Best,
Erick

On Tue, Mar 14, 2017 at 9:23 AM, Elodie Sannier
<elodie.sannier@kelkoo.fr> wrote:
> The request close() method decrements the reference count on the searcher.
>
> public abstract class SolrQueryRequestBase implements SolrQueryRequest,
> Closeable {
>
>   // The index searcher associated with this request
>   protected RefCounted<SolrIndexSearcher> searcherHolder;
>
>   public void close() {
>     if(this.searcherHolder != null) {
>       this.searcherHolder.decref();
>       this.searcherHolder = null;
>     }
>   }
> }
>
> RefCounted keeps track of a reference count on the searcher and closes
> it when the count hits zero.
>
> public abstract class RefCounted<Type> {
>   ...
>   public void decref() {
>     if (refcount.decrementAndGet() == 0) {
>       close();
>     }
>   }
> }
>
> We asume that when we call req.getSearcher() - this increases the
> reference count, after we are done with the searcher, we have to call
> close() to call decref() to decrease the reference count.
>
> But it does not seem enough or maybe there is a bug in Solr in this case ?
>
> Elodie
>
> On 03/14/2017 03:02 PM, Shawn Heisey wrote:
>>
>> On 3/14/2017 3:08 AM, Gerald Reinhart wrote:
>>>
>>> Hi,
>>>     The custom code we have is something like this :
>>> public class MySearchHandlerextends SearchHandler {
>>> @Override public void handleRequestBody(SolrQueryRequest req,
>>> SolrQueryResponse rsp)throws Exception {
>>>          SolrIndexSearcher  searcher =req.getSearcher();
>>>          try{
>>>                   // Do stuff with the searcher....
>>>          }finally {
>>>              req.close();
>>>          }
>>
>> <snip>
>>>
>>>       Despite the fact that we always close the request each time we get
>>> a SolrIndexSearcher from the request, the number of SolrIndexSearcher
>>> instances is increasing. Each time a new commit is done on the index, a
>>> new Searcher is created (this is normal) but the old one remains.     Is
>>> there something wrong with this custom code ?
>>
>> My understanding of Solr and Lucene internals is rudimentary, but I
>> might know what's happening here.
>>
>> The code closes the request, but never closes the searcher.  Searcher
>> objects include a Lucene object that holds onto the index files that
>> pertain to that view of the index.  The searcher must be closed.
>>
>> It does look like if you close the searcher and then close the request,
>> that might be enough to fully decrement all the reference counters
>> involved, but I do not know the code well enough to be sure of that.
>>
>> Thanks,
>> Shawn
>>
>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à l'attention
> exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
> message, merci de le détruire et d'en avertir l'expéditeur.

Mime
View raw message