lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Muir <rcm...@gmail.com>
Subject Re: ThreadPool threads leaking to suite scope.
Date Fri, 02 Mar 2012 10:58:40 GMT
Without looking at the code heavily, I think the problem here involves
wrapped readers like slowMultiReaderWrapper.

Take a look at this code in branch-3x. you will notice that we *never*
both wrap with Slow and add an executor service.

But the real problem is probably that newSearcher should not wrap
readers at all.
instead I suppose tests should explicitly separately call reader =
maybeWrapReader(reader) or something like that, so that they actually
close() the wrapped reader, (not just its inner children).

This will ensure the close hook is actually called... again the issue
is that the test never closes the slow-wrapped reader, because its not
even aware of its existence.

On Fri, Mar 2, 2012 at 5:22 AM, Dawid Weiss
<dawid.weiss@cs.put.poznan.pl> wrote:
> This leak is really frequent if you preallocate threads on the
> execution service. This fragment:
>
>       r.addReaderClosedListener(new ReaderClosedListener() {
>         @Override
>         public void onClose(IndexReader reader) {
>           System.out.println("S:" + ex);
>           shutdownExecutorService(ex);
>         }
>       });
>
> subscribes to the on-close event. I don't see how these listeners can
> be properly called in case of something like
> SlowCompositeReaderWrapper -- it accepts listeners and subscribes them
> to itself but never does anything in doClose. The listener should be
> subscribed to the delegate (I think?) but then the method to add
> listeners is final up the class hierarchy.
>
> ?
>
> Dawid
>
>
> On Thu, Mar 1, 2012 at 12:20 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> Hi,
>>
>> Yeah it's strange. I checked the code: It either does wrap and runs single-thread
searches or it does *not* wrap and runs several threads. So theoretically it should work correctly...
We have to check, if all IndexReaders are correctly closed and the listeners are called.
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>>> -----Original Message-----
>>> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of
>>> Dawid Weiss
>>> Sent: Thursday, March 01, 2012 12:02 PM
>>> To: dev@lucene.apache.org
>>> Subject: Re: ThreadPool threads leaking to suite scope.
>>>
>>> I don't know how to fix it, Uwe, but I know there's definitely something not
all
>>> right with it because threads just keep accumulating (as new searchers are
>>> created).
>>>
>>> I've pushed a static seed for which this is repeatable; this is a heavily worked-on
>>> branch but it may lead you to how to fix this:
>>>
>>> git clone git://github.com/dweiss/lucene_solr.git
>>> git checkout 935e1e9e9a350d6b35b23c4545caf78e82b42747
>>>
>>> try to run TestPhraseQuery (you'll need -ea in Eclipse).
>>>
>>> Dawid
>>>
>>> On Thu, Mar 1, 2012 at 11:55 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
>>> > I think the problem in newSearcher ist hat sometimes the reader is wrapped.
>>> If its wrapped, the underlying reader is only closed, not the wrapper. But the
>>> listener is added to the wrapper. We should add the listener to the original
>>> inner reader.
>>> >
>>> > Uwe
>>> >
>>> > -----
>>> > Uwe Schindler
>>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>>> > http://www.thetaphi.de
>>> > eMail: uwe@thetaphi.de
>>> >
>>> >
>>> >> -----Original Message-----
>>> >> From: Dawid Weiss [mailto:dawid.weiss@gmail.com]
>>> >> Sent: Thursday, March 01, 2012 11:51 AM
>>> >> To: dev@lucene.apache.org
>>> >> Subject: Re: ThreadPool threads leaking to suite scope.
>>> >>
>>> >> > 1) initialize threads eagerly; use ThreadPoolExecutor and call
>>> >> > prestartAllCoreThreads. this could be applied to LTC on the trunk.
>>> >>
>>> >> I did this but threads still leak out from unclosed readers created
>>> >> by LTC#newSearcher. I don't know why, but this isn't called --
>>> >>
>>> >>        r.addReaderClosedListener(new ReaderClosedListener() {
>>> >>          @Override
>>> >>          public void onClose(IndexReader reader) {
>>> >>            shutdownExecutorService(ex);
>>> >>          }
>>> >>        });
>>> >>
>>> >> Clues?
>>> >>
>>> >> Dawid
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>>> >> additional commands, e-mail: dev-help@lucene.apache.org
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>>> > additional commands, e-mail: dev-help@lucene.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>>> commands, e-mail: dev-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message