lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Reure <jbre...@weborganic.com>
Subject Re: Question about open files
Date Tue, 05 Apr 2011 23:59:17 GMT
We are using version 3.0.3. So you can confirm that closing the writer (and
the reader created from that writer) should be enough to release the file
handles?
If that is the case, that means our application has a bug somewhere that i
need to track down.
Thanks,
JB.

On 5 April 2011 19:48, Ian Lea <ian.lea@gmail.com> wrote:

> Which version of lucene are you using?  Something changed in the 3.x
> release, and maybe 2.9.x, in the way that old file handles are closed.
>  Previously it wasn't always necessary to explicitly close everything,
> now it is.
>
> Your usage sounds fine to me.  When I've hit too many open files when
> using 3.0.x it has always been because something, somewhere, wasn't
> being closed.
>
>
> --
> Ian.
>
>
> On Tue, Apr 5, 2011 at 8:49 AM, Jean-Baptiste Reure
> <jbreure@weborganic.com> wrote:
> > Hi all,
> >
> > I have been looking for information about this and found a few things
> here
> > and there but nothing very clear on when files are opened and closed by
> > Lucene.
> >
> > We have an application that uses Lucene quite heavily in the following
> > fashion: there are multiple indexes in use at all times. For each index,
> a
> > single IndexWriter object is constantly opened (and used to update the
> > index) and a single IndexReader is also constantly opened. This Reader is
> > created using the almost real-time reader of the IndexWriter. The Reader
> is
> > used to query the index and when the Writer updates an index, the
> > corresponding reader is re-opened and the previous one is discarded when
> all
> > searches using it are finished. Periodically, the index is optimised if
> it
> > needs to be (if the writer has been used).
> >
> > This is all working perfectly fine but we've been running into an issue
> > recently when we've add a lot of indexes on one server (more than a 1000
> > indexes). This led to the "Too many open files" exception, so we
> increased
> > the number of open files allowed on that server but obviously, it's a
> > temporary solution.
> >
> > So we decided to implement a mechanism that would keep only a maximum
> number
> > of indexes "open" at all times. When the maximum number is reached, the
> > least recently used indexes are closed (closing the IndexWriter and the
> > IndexReader), and if these indexes are needed again in the future, they
> are
> > "re-opened".
> > This is functioning fine as well, as we can see ".lock" files appearing
> and
> > disappearing in the index directories, but handles on the segment files
> are
> > still in use.
> >
> > So my question is when are these file handles discarded? I was under the
> > impression from reading the documentation that closing an IndexReader was
> > enough to close the files associated with that reader but we went even
> > further by closing the IndexWriter as well but to no avail. If it should
> > work that way, it means we have a bug somewhere with readers not being
> > closed properly but I couldn't find one last time I checked.
> >
> > Also, do you think it is the right way to go around the "too many open
> > files" issue? Can you recommend using another method?
> >
> > Thanks for any help.
> >
> > JB Reure
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message