lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Question about open files
Date Tue, 05 Apr 2011 09:48:55 GMT
Which version of lucene are you using?  Something changed in the 3.x
release, and maybe 2.9.x, in the way that old file handles are closed.
 Previously it wasn't always necessary to explicitly close everything,
now it is.

Your usage sounds fine to me.  When I've hit too many open files when
using 3.0.x it has always been because something, somewhere, wasn't
being closed.


--
Ian.


On Tue, Apr 5, 2011 at 8:49 AM, Jean-Baptiste Reure
<jbreure@weborganic.com> wrote:
> Hi all,
>
> I have been looking for information about this and found a few things here
> and there but nothing very clear on when files are opened and closed by
> Lucene.
>
> We have an application that uses Lucene quite heavily in the following
> fashion: there are multiple indexes in use at all times. For each index, a
> single IndexWriter object is constantly opened (and used to update the
> index) and a single IndexReader is also constantly opened. This Reader is
> created using the almost real-time reader of the IndexWriter. The Reader is
> used to query the index and when the Writer updates an index, the
> corresponding reader is re-opened and the previous one is discarded when all
> searches using it are finished. Periodically, the index is optimised if it
> needs to be (if the writer has been used).
>
> This is all working perfectly fine but we've been running into an issue
> recently when we've add a lot of indexes on one server (more than a 1000
> indexes). This led to the "Too many open files" exception, so we increased
> the number of open files allowed on that server but obviously, it's a
> temporary solution.
>
> So we decided to implement a mechanism that would keep only a maximum number
> of indexes "open" at all times. When the maximum number is reached, the
> least recently used indexes are closed (closing the IndexWriter and the
> IndexReader), and if these indexes are needed again in the future, they are
> "re-opened".
> This is functioning fine as well, as we can see ".lock" files appearing and
> disappearing in the index directories, but handles on the segment files are
> still in use.
>
> So my question is when are these file handles discarded? I was under the
> impression from reading the documentation that closing an IndexReader was
> enough to close the files associated with that reader but we went even
> further by closing the IndexWriter as well but to no avail. If it should
> work that way, it means we have a bug somewhere with readers not being
> closed properly but I couldn't find one last time I checked.
>
> Also, do you think it is the right way to go around the "too many open
> files" issue? Can you recommend using another method?
>
> Thanks for any help.
>
> JB Reure
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message