lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: File Handle Leaks During Lucene 3.0.2 Merge
Date Fri, 01 Oct 2010 06:00:04 GMT
Hi Jamie,

YES, ist expected for the reasons described above (segments are still
referenced by the open IndexReaders, but files were already deleted by
IndexWriter). The approx. number of open, but already deleted files should
be approx. stable.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Jamie [mailto:jamie@stimulussoft.com]
> Sent: Friday, October 01, 2010 7:41 AM
> To: java-user@lucene.apache.org
> Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge
> 
>   Hi Mike
> 
>   I managed to get hold of a copy of your book through Safari Books.
> Quite an impressive online reading system they have there! I integrated
your
> SearchManager class into our code, but I am still seeing file handles
marked
> deleted in the index directory. I am running the following command on
Linux:
> 
> sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
> 
> Every 0.1s: lsof | grep /var/index | grep deleted |...  Fri Oct  1
> 09:37:36 2010
> 
> 54
> 
> The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32,
etc. They
> seem stable though. Is this to be expected when using NRT search?
> 
>   I am pretty certain that all Searchers are released at the end of every
search.
> I double checked it at least twenty times.
> 
> Jamie
> 
> 
> 
> On 2010/09/30 11:56 PM, Michael McCandless wrote:
> > On Thu, Sep 30, 2010 at 5:59 AM, Jamie<jamie@stimulussoft.com>  wrote:
> >>   Hi Michael / Uwe
> >>
> >>> It's good to cache the reader, but, finalize would worry me too
> >>> since you have no control over when GC gets around to calling it...
> >>> you risk tying up resources for longer than necessary.
> >> I did it this way, as I didn't want to over complicate the code by
> >> introducing mechanisms to track the number of search threads using a
> >> shared indexreader. Admittedly, its not a very clean solution but in
> >> my case it does work. Is there a particular technique for knowing
> >> when to a close a reader when there are multiple search threads using
> >> that reader? Should I keep some kind of counter and override the
> >> close method of the reader such that the underlying reader is only
closed
> when everyone's done with it?
> > See Uwe's response (or SearcherManager).
> >
> >>> IndexWriter has a reader pool, internally, where it holds open
> >>> SegmentReaders for the still-live segments in the index.  This is
> >>> used by IndexReader.reopen to share open SegmentReaders.
> >>>
> >>> But the open files should correspond only to segments still "live"
> >>> in the index.  After segments are merged away, these readers are
dropped.
> >>>   Is this what you are seeing?
> >>>
> >> I dont fully understand your explanation/question. When I run lsof, I
> >> am seeing the following:
> >>
> >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cf
> >> s
> >> (deleted)
> >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cf
> >> s
> >> (deleted)
> >>
> >> I assume these are left by the OS after the merge operation tried to
> >> delete old segments. The OS is unable to delete the files. I think
> >> its because our new code never closes the indexwriter, but rather
> >> uses the
> >> indexwriter.commit() method to apply the changes. Is this correct?
> > Ahh I see they are deleted but held open... hmmm.
> >
> > Though this is also what you'd see if there were still a reader open.
> > Are you certain all readers were closed (finalized) when you ran lsof?
> >
> > Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message