lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Question about open files
Date Wed, 06 Apr 2011 12:13:17 GMT
I suspect you're already aware of this, but I've
overlooked the obvious so many times I thought
I'd mention it...

A classic mistake is to assign a reader with reopen
and not close the old reader, see:
http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/index/IndexReader.html#reopen()

<http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/index/IndexReader.html#reopen()>
Erick.

On Tue, Apr 5, 2011 at 7:59 PM, Jean-Baptiste Reure
<jbreure@weborganic.com>wrote:

> We are using version 3.0.3. So you can confirm that closing the writer (and
> the reader created from that writer) should be enough to release the file
> handles?
> If that is the case, that means our application has a bug somewhere that i
> need to track down.
> Thanks,
> JB.
>
> On 5 April 2011 19:48, Ian Lea <ian.lea@gmail.com> wrote:
>
> > Which version of lucene are you using?  Something changed in the 3.x
> > release, and maybe 2.9.x, in the way that old file handles are closed.
> >  Previously it wasn't always necessary to explicitly close everything,
> > now it is.
> >
> > Your usage sounds fine to me.  When I've hit too many open files when
> > using 3.0.x it has always been because something, somewhere, wasn't
> > being closed.
> >
> >
> > --
> > Ian.
> >
> >
> > On Tue, Apr 5, 2011 at 8:49 AM, Jean-Baptiste Reure
> > <jbreure@weborganic.com> wrote:
> > > Hi all,
> > >
> > > I have been looking for information about this and found a few things
> > here
> > > and there but nothing very clear on when files are opened and closed by
> > > Lucene.
> > >
> > > We have an application that uses Lucene quite heavily in the following
> > > fashion: there are multiple indexes in use at all times. For each
> index,
> > a
> > > single IndexWriter object is constantly opened (and used to update the
> > > index) and a single IndexReader is also constantly opened. This Reader
> is
> > > created using the almost real-time reader of the IndexWriter. The
> Reader
> > is
> > > used to query the index and when the Writer updates an index, the
> > > corresponding reader is re-opened and the previous one is discarded
> when
> > all
> > > searches using it are finished. Periodically, the index is optimised if
> > it
> > > needs to be (if the writer has been used).
> > >
> > > This is all working perfectly fine but we've been running into an issue
> > > recently when we've add a lot of indexes on one server (more than a
> 1000
> > > indexes). This led to the "Too many open files" exception, so we
> > increased
> > > the number of open files allowed on that server but obviously, it's a
> > > temporary solution.
> > >
> > > So we decided to implement a mechanism that would keep only a maximum
> > number
> > > of indexes "open" at all times. When the maximum number is reached, the
> > > least recently used indexes are closed (closing the IndexWriter and the
> > > IndexReader), and if these indexes are needed again in the future, they
> > are
> > > "re-opened".
> > > This is functioning fine as well, as we can see ".lock" files appearing
> > and
> > > disappearing in the index directories, but handles on the segment files
> > are
> > > still in use.
> > >
> > > So my question is when are these file handles discarded? I was under
> the
> > > impression from reading the documentation that closing an IndexReader
> was
> > > enough to close the files associated with that reader but we went even
> > > further by closing the IndexWriter as well but to no avail. If it
> should
> > > work that way, it means we have a bug somewhere with readers not being
> > > closed properly but I couldn't find one last time I checked.
> > >
> > > Also, do you think it is the right way to go around the "too many open
> > > files" issue? Can you recommend using another method?
> > >
> > > Thanks for any help.
> > >
> > > JB Reure
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message