Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4CA57479.8000405@stimulussoft.com>
Date: Fri, 01 Oct 2010 07:41:13 +0200
From: Jamie <jamie@stimulussoft.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
 rv:1.9.2.9) Gecko/20100915 Thunderbird/3.1.4
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge
References: <4CA37BD8.8050608@stimulussoft.com>
	<006d01cb6003$2f298650$8d7c92f0$@thetaphi.de>
	<4CA38A5F.1000100@stimulussoft.com>
	<000001cb607d$486b9660$d942c320$@thetaphi.de>
	<4CA457AE.8080203@stimulussoft.com>
	<AANLkTin43PLTQKt5oLqd9GreaqUdnrR31cumF9BpBtnD@mail.gmail.com>
	<4CA45F93.4000503@stimulussoft.com>
 <AANLkTikWz0rG3YfEbX_3Og4vD=b8TaW81hGn-RhqXHwe@mail.gmail.com>
In-Reply-To: <AANLkTikWz0rG3YfEbX_3Og4vD=b8TaW81hGn-RhqXHwe@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

  Hi Mike

  I managed to get hold of a copy of your book through Safari Books. 
Quite an impressive online reading system they have there! I integrated 
your SearchManager class into our code, but I am still seeing file 
handles marked deleted in the index directory. I am running the 
following command on Linux:

sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"

Every 0.1s: lsof | grep /var/index | grep deleted |...  Fri Oct  1 
09:37:36 2010

54

The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, 
etc. They seem stable though. Is this to be expected when using NRT search?

  I am pretty certain that all Searchers are released at the end of 
every search. I double checked it at least twenty times.

Jamie


On 2010/09/30 11:56 PM, Michael McCandless wrote:
> On Thu, Sep 30, 2010 at 5:59 AM, Jamie<jamie@stimulussoft.com>  wrote:
>>   Hi Michael / Uwe
>>
>>> It's good to cache the reader, but, finalize would worry me too since
>>> you have no control over when GC gets around to calling it... you risk
>>> tying up resources for longer than necessary.
>> I did it this way, as I didn't want to over complicate the code by
>> introducing mechanisms to track the number of search threads using a shared
>> indexreader. Admittedly, its not a very clean solution but in my case it
>> does work. Is there a particular technique for knowing when to a close a
>> reader when there are multiple search threads using that reader? Should I
>> keep some kind of counter and override the close method of the reader such
>> that the underlying reader is only closed when everyone's done with it?
> See Uwe's response (or SearcherManager).
>
>>> IndexWriter has a reader pool, internally, where it holds open
>>> SegmentReaders for the still-live segments in the index.  This is used
>>> by IndexReader.reopen to share open SegmentReaders.
>>>
>>> But the open files should correspond only to segments still "live" in
>>> the index.  After segments are merged away, these readers are dropped.
>>>   Is this what you are seeing?
>>>
>> I dont fully understand your explanation/question. When I run lsof, I am
>> seeing the following:
>>
>> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cfs
>> (deleted)
>> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cfs
>> (deleted)
>>
>> I assume these are left by the OS after the merge operation tried to delete
>> old segments. The OS is unable to delete the files. I think its because our
>> new code never closes the indexwriter, but rather uses the
>> indexwriter.commit() method to apply the changes. Is this correct?
> Ahh I see they are deleted but held open... hmmm.
>
> Though this is also what you'd see if there were still a reader open.
> Are you certain all readers were closed (finalized) when you ran lsof?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org