lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Reure <jbre...@weborganic.com>
Subject Re: Question about open files
Date Thu, 07 Apr 2011 05:40:36 GMT
I found what the problem was: we were closing the IndexSearcher but not the
underlying IndexReader (I wrongly assumed that closing one would close the
other).
Everything is working perfectly now, thanks for the help.

JB

On 6 April 2011 22:13, Erick Erickson <erickerickson@gmail.com> wrote:

> I suspect you're already aware of this, but I've
> overlooked the obvious so many times I thought
> I'd mention it...
>
> A classic mistake is to assign a reader with reopen
> and not close the old reader, see:
>
> http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/index/IndexReader.html#reopen()
>
> <
> http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/index/IndexReader.html#reopen()
> >
> Erick.
>
> On Tue, Apr 5, 2011 at 7:59 PM, Jean-Baptiste Reure
> <jbreure@weborganic.com>wrote:
>
> > We are using version 3.0.3. So you can confirm that closing the writer
> (and
> > the reader created from that writer) should be enough to release the file
> > handles?
> > If that is the case, that means our application has a bug somewhere that
> i
> > need to track down.
> > Thanks,
> > JB.
> >
> > On 5 April 2011 19:48, Ian Lea <ian.lea@gmail.com> wrote:
> >
> > > Which version of lucene are you using?  Something changed in the 3.x
> > > release, and maybe 2.9.x, in the way that old file handles are closed.
> > >  Previously it wasn't always necessary to explicitly close everything,
> > > now it is.
> > >
> > > Your usage sounds fine to me.  When I've hit too many open files when
> > > using 3.0.x it has always been because something, somewhere, wasn't
> > > being closed.
> > >
> > >
> > > --
> > > Ian.
> > >
> > >
> > > On Tue, Apr 5, 2011 at 8:49 AM, Jean-Baptiste Reure
> > > <jbreure@weborganic.com> wrote:
> > > > Hi all,
> > > >
> > > > I have been looking for information about this and found a few things
> > > here
> > > > and there but nothing very clear on when files are opened and closed
> by
> > > > Lucene.
> > > >
> > > > We have an application that uses Lucene quite heavily in the
> following
> > > > fashion: there are multiple indexes in use at all times. For each
> > index,
> > > a
> > > > single IndexWriter object is constantly opened (and used to update
> the
> > > > index) and a single IndexReader is also constantly opened. This
> Reader
> > is
> > > > created using the almost real-time reader of the IndexWriter. The
> > Reader
> > > is
> > > > used to query the index and when the Writer updates an index, the
> > > > corresponding reader is re-opened and the previous one is discarded
> > when
> > > all
> > > > searches using it are finished. Periodically, the index is optimised
> if
> > > it
> > > > needs to be (if the writer has been used).
> > > >
> > > > This is all working perfectly fine but we've been running into an
> issue
> > > > recently when we've add a lot of indexes on one server (more than a
> > 1000
> > > > indexes). This led to the "Too many open files" exception, so we
> > > increased
> > > > the number of open files allowed on that server but obviously, it's a
> > > > temporary solution.
> > > >
> > > > So we decided to implement a mechanism that would keep only a maximum
> > > number
> > > > of indexes "open" at all times. When the maximum number is reached,
> the
> > > > least recently used indexes are closed (closing the IndexWriter and
> the
> > > > IndexReader), and if these indexes are needed again in the future,
> they
> > > are
> > > > "re-opened".
> > > > This is functioning fine as well, as we can see ".lock" files
> appearing
> > > and
> > > > disappearing in the index directories, but handles on the segment
> files
> > > are
> > > > still in use.
> > > >
> > > > So my question is when are these file handles discarded? I was under
> > the
> > > > impression from reading the documentation that closing an IndexReader
> > was
> > > > enough to close the files associated with that reader but we went
> even
> > > > further by closing the IndexWriter as well but to no avail. If it
> > should
> > > > work that way, it means we have a bug somewhere with readers not
> being
> > > > closed properly but I couldn't find one last time I checked.
> > > >
> > > > Also, do you think it is the right way to go around the "too many
> open
> > > > files" issue? Can you recommend using another method?
> > > >
> > > > Thanks for any help.
> > > >
> > > > JB Reure
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message