lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject RE: References to deleted file handles in long-running server application
Date Fri, 18 Nov 2005 18:21:26 GMT
I was thinking of doing something like this for my numerous indices
over on Simpy.com.  However, I was thinking of something much simpler -
an IndexSearcher subclass with close() overriden.  The implementation
of close() would then just start a Thread and call super.close() after
N seconds.

I haven't done this yet, so I'm not sure if this will do, but it seems
simpler than the contribution in JIRA.

Otis

--- "Vanlerberghe, Luc" <Luc.Vanlerberghe@bvdep.com> wrote:

> Good rule of thumb: don't ever count on the garbage collector
> cleaning
> up for you (even if you call System.gc() to give it a hint).
> 
> You should close your IndexSearchers, but with a multithreaded
> application  it's difficult to know when (you have to keep them open
> until no thread uses it any more)
> 
> I created a class DelayCloseIndexSearcher class and posted it in jira
> just for this purpose (See
> http://issues.apache.org/jira/browse/LUCENE-445)
> There's a use case in the javadoc.
> 
> I'm using it in my production system (but in a more complex way than
> in
> the example code) and I don't (seem to :) have any problems...
> 
> Either way, I would very much appreciate some feedback about the
> code.
> 
> Thanks,
> 
> Luc
> 
> -----Original Message-----
> From: Matt Magoffin [mailto:apache.org@msqr.us] 
> Sent: vrijdag 18 november 2005 8:58
> To: java-user@lucene.apache.org
> Subject: RE: References to deleted file handles in long-running
> server
> application
> 
> I'm updating nearly continuously (probably average about every 10
> seconds). I don't explicitly close the IndexSearcher objects I
> create,
> as
> I share them across threads, but do leave them to be garbage
> collected.
> I
> ran into index corruption issues when I explicitly closed them, since
> I
> don't have any code keeping track of how many threads might be using
> a
> given IndexSearcher.
> 
> I wonder if it's worth the effort of adding that code in, or can I be
> ensured that when the IndexSearcher objects are garbage collected the
> file
> handles will be released?
> 
> -- m@
> 
> > How often are you updating your index?  Are you closing your old
> > IndexSearchers after switching over to the new index?  You'll need
> to
> > close the searchers in order to release the file handle.  This was
> the
> > same issue I was experiencing:
> >
> >
>
http://mail-archives.apache.org/mod_mbox/lucene-java-user/200504.mbox/%3
> > c200504281819640.SM01836@chicago%3e
> >
> > - Monsur
> >
> >
> >
> >> -----Original Message-----
> >> From: Matt Magoffin [mailto:apache.org@msqr.us]
> >> Sent: Thursday, November 17, 2005 11:05 PM
> >> To: java-user@lucene.apache.org
> >> Subject: Re: References to deleted file handles in
> >> long-running server application
> >>
> >>
> >> I've been watching our servers today, and now there are 2500
> >> "deleted" file handles open like this. Seems to be quite
> >> large. Still don't know why there are so many. I'm using the
> >> compound index format already to reduce the number of open files.
> >>
> >> -- m@
> >>
> >> > Hello, I use Lucene in a long-running server application on a
> Linux
> >> > server, and the other day I got the "Too many open files"
> >> exception.
> >> > I've increased the number of allowed file handles, but was
> checking
> >> > out the open file handles using "lsof", and see about 300
> >> files listed
> >> > like the
> >> > following:
> >> >
> >> > java       1458  jboss  436r   REG        8,2      3945  
> 6258825
> >> > /var/lucene-index/_1o6hl.cfs (deleted)
> >> > java       1458  jboss  437r   REG        8,2      3467  
> 6258829
> >> > /var/lucene-index/_1o6hp.cfs (deleted)
> >> > java       1458  jboss  438r   REG        8,2      2743  
> 6258826
> >> > /var/lucene-index/_1o6ht.cfs (deleted)
> >> > java       1458  jboss  439r   REG        8,2      4069  
> 6258817
> >> > /var/lucene-index/_1o6hx.cfs (deleted)
> >> > java       1458  jboss  440r   REG        8,2      4098  
> 6258830
> >> > /var/lucene-index/_1o6i1.cfs (deleted)
> >> > java       1458  jboss  441r   REG        8,2      1817  
> 6258821
> >> > /var/lucene-index/_1o6i3.cfs (deleted)
> >> >
> >> > I haven't been monitoring this long enough to tell if the number
> >> > simply continues to grow over time or levels off at some point.
> >> >
> >> > I was wondering if this behavior to be expected? When will the
> >> > application release these deleted file handles (if ever)?
> Perhaps
> >> > during garbage collection? Or might I be handling the index
> >> > incorrectly in some way?
> >> >
> >> > Any thoughts appreciated.
> >> > -- m@
> >>
> >>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message