Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of trewig@mufin.com designates
 195.214.216.122 as permitted sender)
Message-ID: <4CDA7F86.4090002@mufin.com>
Date: Wed, 10 Nov 2010 12:18:30 +0100
From: Thomas Rewig <trewig@mufin.com>
Organization: mufin
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; de;
 rv:1.9.2.9) Gecko/20100915 Thunderbird/3.1.4
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge
References: <4CA37BD8.8050608@stimulussoft.com>
	<006d01cb6003$2f298650$8d7c92f0$@thetaphi.de>
	<4CA38A5F.1000100@stimulussoft.com>
	<000001cb607d$486b9660$d942c320$@thetaphi.de>
	<4CA457AE.8080203@stimulussoft.com>
	<AANLkTin43PLTQKt5oLqd9GreaqUdnrR31cumF9BpBtnD@mail.gmail.com>
	<4CA45F93.4000503@stimulussoft.com>
	<AANLkTikWz0rG3YfEbX_3Og4vD=b8TaW81hGn-RhqXHwe@mail.gmail.com>
	<4CA57479.8000405@stimulussoft.com>
 <AANLkTinmP6i5hMwTAiZPOb221_ieOVNon3sQGThyUiRK@mail.gmail.com>
In-Reply-To: <AANLkTinmP6i5hMwTAiZPOb221_ieOVNon3sQGThyUiRK@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------040909080000020108070503"

--------------040909080000020108070503
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

  Hello,
please excuse that I hijack this old thread but I have the same problem 
with the deleted file handles, so I think this is the right place for.

I also integrated the searchManager in our Code and see the file handles 
fluctuate up and down. At first glance the situation seems stable but 
the count of the deleted file handles never drops to zero and I think 
the deleted file handles are constantly rising. About +50 handles stay 
per day by refreshing the searchers every 2 minutes and committing after 
10 minutes. I think some deleted handles remain open after the 
committing, but I could not figure out why. The max value of deleted 
handles until now was about 250, because the Index will be changed after 
a few days. Jamie could you achieve that all deleted handles have been 
removed? Do you have some information how to repair this or is that an 
expected behavior and the amount of the deleted file handles will be 
stable - but first at a amount of 500 or so.

Thanks in advance

Thomas

>> I integrated your
>> SearchManager class into our code, but I am still seeing file handles marked
>> deleted in the index directory. I am running the following command on Linux:
>>
>> sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
>>
>> Every 0.1s: lsof | grep /var/index | grep deleted |...  Fri Oct  1 09:37:36
>> 2010
>>
>> 54
>>
>> The deleted file handles fluctuate up and down. 54 ->  102 ->  64 ->  32, etc.
>> They seem stable though. Is this to be expected when using NRT search?
>>
>>   I am pretty certain that all Searchers are released at the end of every
>> search. I double checked it at least twenty times.
> SearcherManager always keeps one IndexReader open (the current "live"
> one), so it's expected that this reader is holding open references to
> deleted files if in fact IndexWriter has completed merges since the
> reader was opened. Though.. I would expect you to sometimes see 0
> deleted files in your lsof...
>
> Maybe try this: 1) pause your indexing, 2) reopen the reader (call
> SearcherManager.reopen), and then 3) run lsof again, at which point
> you should see no deleted files still held open.
>
> But note that to truly pause your indexing, you should switch
> (temporarily, for this test) your IndexWriter to the
> SerialMergeScheduler, and then ensure you stop calling IndexWriter ops
> while you run steps 2 and 3.
>
> Mike
>


--------------040909080000020108070503--