lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Question about open files
Date Wed, 06 Apr 2011 08:53:06 GMT
Yes, to the best of my knowledge and experience, closing readers and
writers releases the file handles.


--
Ian.


On Wed, Apr 6, 2011 at 12:59 AM, Jean-Baptiste Reure
<jbreure@weborganic.com> wrote:
> We are using version 3.0.3. So you can confirm that closing the writer (and
> the reader created from that writer) should be enough to release the file
> handles?
> If that is the case, that means our application has a bug somewhere that i
> need to track down.
> Thanks,
> JB.
>
> On 5 April 2011 19:48, Ian Lea <ian.lea@gmail.com> wrote:
>
>> Which version of lucene are you using?  Something changed in the 3.x
>> release, and maybe 2.9.x, in the way that old file handles are closed.
>>  Previously it wasn't always necessary to explicitly close everything,
>> now it is.
>>
>> Your usage sounds fine to me.  When I've hit too many open files when
>> using 3.0.x it has always been because something, somewhere, wasn't
>> being closed.
>>
>>
>> --
>> Ian.
>>
>>
>> On Tue, Apr 5, 2011 at 8:49 AM, Jean-Baptiste Reure
>> <jbreure@weborganic.com> wrote:
>> > Hi all,
>> >
>> > I have been looking for information about this and found a few things
>> here
>> > and there but nothing very clear on when files are opened and closed by
>> > Lucene.
>> >
>> > We have an application that uses Lucene quite heavily in the following
>> > fashion: there are multiple indexes in use at all times. For each index,
>> a
>> > single IndexWriter object is constantly opened (and used to update the
>> > index) and a single IndexReader is also constantly opened. This Reader is
>> > created using the almost real-time reader of the IndexWriter. The Reader
>> is
>> > used to query the index and when the Writer updates an index, the
>> > corresponding reader is re-opened and the previous one is discarded when
>> all
>> > searches using it are finished. Periodically, the index is optimised if
>> it
>> > needs to be (if the writer has been used).
>> >
>> > This is all working perfectly fine but we've been running into an issue
>> > recently when we've add a lot of indexes on one server (more than a 1000
>> > indexes). This led to the "Too many open files" exception, so we
>> increased
>> > the number of open files allowed on that server but obviously, it's a
>> > temporary solution.
>> >
>> > So we decided to implement a mechanism that would keep only a maximum
>> number
>> > of indexes "open" at all times. When the maximum number is reached, the
>> > least recently used indexes are closed (closing the IndexWriter and the
>> > IndexReader), and if these indexes are needed again in the future, they
>> are
>> > "re-opened".
>> > This is functioning fine as well, as we can see ".lock" files appearing
>> and
>> > disappearing in the index directories, but handles on the segment files
>> are
>> > still in use.
>> >
>> > So my question is when are these file handles discarded? I was under the
>> > impression from reading the documentation that closing an IndexReader was
>> > enough to close the files associated with that reader but we went even
>> > further by closing the IndexWriter as well but to no avail. If it should
>> > work that way, it means we have a bug somewhere with readers not being
>> > closed properly but I couldn't find one last time I checked.
>> >
>> > Also, do you think it is the right way to go around the "too many open
>> > files" issue? Can you recommend using another method?
>> >
>> > Thanks for any help.
>> >
>> > JB Reure
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message