lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Smith <psm...@aconex.com>
Subject Re: References to deleted file handles in long-running server application
Date Fri, 18 Nov 2005 08:04:22 GMT
I have found exactly the same problem.  You should consider adding an  
'expired" IndexSearcher to some sort of Buffer that closes the  
IndexSearcher after some default timeout.  I have not found in  
practice that allowing the GC to do the work does quite what you  
think it might do (particularly on Linux).

  It could be to do with the different constructors of IndexSearcher  
(String vs IndexReader), but my advise is to mark the IndexSearcher  
as ready to be closed by placing it in a buffer, and after some  
sensible timeout where you would consider any existing queries that  
may have been running at the time of 'close' should be cancelled by  
force closing the IndexSearcher.  If you were to do this every X  
minutes, then any running Searcher that was still being used after  
this time is probably a candidate for an interrupt.  (In a web  
search, the user has probably navigated away and given up on the long  
running search anyway).

Paul Smith

On 18/11/2005, at 6:57 PM, Matt Magoffin wrote:

> I'm updating nearly continuously (probably average about every 10
> seconds). I don't explicitly close the IndexSearcher objects I  
> create, as
> I share them across threads, but do leave them to be garbage  
> collected. I
> ran into index corruption issues when I explicitly closed them,  
> since I
> don't have any code keeping track of how many threads might be using a
> given IndexSearcher.
>
> I wonder if it's worth the effort of adding that code in, or can I be
> ensured that when the IndexSearcher objects are garbage collected  
> the file
> handles will be released?
>
> -- m@
>
>> How often are you updating your index?  Are you closing your old
>> IndexSearchers after switching over to the new index?  You'll need to
>> close the searchers in order to release the file handle.  This was  
>> the
>> same issue I was experiencing:
>>
>> http://mail-archives.apache.org/mod_mbox/lucene-java-user/ 
>> 200504.mbox/%3
>> c200504281819640.SM01836@chicago%3e
>>
>> - Monsur
>>
>>
>>
>>> -----Original Message-----
>>> From: Matt Magoffin [mailto:apache.org@msqr.us]
>>> Sent: Thursday, November 17, 2005 11:05 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Re: References to deleted file handles in
>>> long-running server application
>>>
>>>
>>> I've been watching our servers today, and now there are 2500
>>> "deleted" file handles open like this. Seems to be quite
>>> large. Still don't know why there are so many. I'm using the
>>> compound index format already to reduce the number of open files.
>>>
>>> -- m@
>>>
>>>> Hello, I use Lucene in a long-running server application on a Linux
>>>> server, and the other day I got the "Too many open files"
>>> exception.
>>>> I've increased the number of allowed file handles, but was checking
>>>> out the open file handles using "lsof", and see about 300
>>> files listed
>>>> like the
>>>> following:
>>>>
>>>> java       1458  jboss  436r   REG        8,2      3945   6258825
>>>> /var/lucene-index/_1o6hl.cfs (deleted)
>>>> java       1458  jboss  437r   REG        8,2      3467   6258829
>>>> /var/lucene-index/_1o6hp.cfs (deleted)
>>>> java       1458  jboss  438r   REG        8,2      2743   6258826
>>>> /var/lucene-index/_1o6ht.cfs (deleted)
>>>> java       1458  jboss  439r   REG        8,2      4069   6258817
>>>> /var/lucene-index/_1o6hx.cfs (deleted)
>>>> java       1458  jboss  440r   REG        8,2      4098   6258830
>>>> /var/lucene-index/_1o6i1.cfs (deleted)
>>>> java       1458  jboss  441r   REG        8,2      1817   6258821
>>>> /var/lucene-index/_1o6i3.cfs (deleted)
>>>>
>>>> I haven't been monitoring this long enough to tell if the number
>>>> simply continues to grow over time or levels off at some point.
>>>>
>>>> I was wondering if this behavior to be expected? When will the
>>>> application release these deleted file handles (if ever)? Perhaps
>>>> during garbage collection? Or might I be handling the index
>>>> incorrectly in some way?
>>>>
>>>> Any thoughts appreciated.
>>>> -- m@
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


Mime
View raw message