lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Reure <jbre...@weborganic.com>
Subject Question about open files
Date Tue, 05 Apr 2011 07:49:00 GMT
Hi all,

I have been looking for information about this and found a few things here
and there but nothing very clear on when files are opened and closed by
Lucene.

We have an application that uses Lucene quite heavily in the following
fashion: there are multiple indexes in use at all times. For each index, a
single IndexWriter object is constantly opened (and used to update the
index) and a single IndexReader is also constantly opened. This Reader is
created using the almost real-time reader of the IndexWriter. The Reader is
used to query the index and when the Writer updates an index, the
corresponding reader is re-opened and the previous one is discarded when all
searches using it are finished. Periodically, the index is optimised if it
needs to be (if the writer has been used).

This is all working perfectly fine but we've been running into an issue
recently when we've add a lot of indexes on one server (more than a 1000
indexes). This led to the "Too many open files" exception, so we increased
the number of open files allowed on that server but obviously, it's a
temporary solution.

So we decided to implement a mechanism that would keep only a maximum number
of indexes "open" at all times. When the maximum number is reached, the
least recently used indexes are closed (closing the IndexWriter and the
IndexReader), and if these indexes are needed again in the future, they are
"re-opened".
This is functioning fine as well, as we can see ".lock" files appearing and
disappearing in the index directories, but handles on the segment files are
still in use.

So my question is when are these file handles discarded? I was under the
impression from reading the documentation that closing an IndexReader was
enough to close the files associated with that reader but we went even
further by closing the IndexWriter as well but to no avail. If it should
work that way, it means we have a bug somewhere with readers not being
closed properly but I couldn't find one last time I checked.

Also, do you think it is the right way to go around the "too many open
files" issue? Can you recommend using another method?

Thanks for any help.

JB Reure

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message