lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josef (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-2762) Don't leak deleted open file handles with pooled readers
Date Tue, 24 May 2011 12:56:47 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038533#comment-13038533
] 

Josef commented on LUCENE-2762:
-------------------------------

Since I am still observing file references on deleted index files, I have a quick question
regarding this issue:

We are using Lucene 3.0.3 with standard configuration and near real-time reader, as the following
pseudo code shows:

{code}
...
        // initially obtaining and referencing a near-real time searcher
        IndexSearcher currentSearcher = new IndexSearcher(writer.getReader())
...

	// subsequently obtaining a new near-real time searcher if necessary
	if (!currentSearcher.getIndexReader().isCurrent()) {
	    IndexReader newReader = currentSearcher.getIndexReader().reopen();
	    IndexSearcher newSearcher = new IndexSearcher(newReader);
	    
	    // release old searcher (by decreasing reference)
	    currentSearcher.getIndexReader().decRef();
	    
	    currentSearcher = newSearcher;
	}
...
{code}

After running the application for a while with index updates and doing new searcher using
the newly obtained IndexSearcher, we noticed that JVM still holds references to already deleted
index files.
For example:
{code}
java    20742 xxx  394r   REG                8,6      366 3412376 /home/xxx/Desktop/mp.home/dev/index/Artist/_fc.cfs
(deleted)
java    20742 xxx  398r   REG                8,6      366 3412375 /home/xxx/Desktop/mp.home/dev/index/Artist/_fq.cfs
java    20742 xxx  401uw  REG                8,6        0 3412333 /home/xxx/Desktop/mp.home/dev/index/Artist/write.lock
java    20742 xxx  415r   REG                8,6      128 3412349 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.tis
java    20742 xxx  416r   REG                8,6      366 3412341 /home/xxx/Desktop/mp.home/dev/index/Artist/_fd.cfs
(deleted)
java    20742 xxx  417r   REG                8,6      366 3412344 /home/xxx/Desktop/mp.home/dev/index/Artist/_fe.cfs
(deleted)
java    20742 xxx  418r   REG                8,6       71 3412356 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.tis
(deleted)
java    20742 xxx  424r   REG                8,6        7 3412362 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.frq
(deleted)
java    20742 xxx  425r   REG                8,6        7 3412363 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.prx
(deleted)
java    20742 xxx  426r   REG                8,6       23 3412351 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.fdt
(deleted)
java    20742 xxx  427r   REG                8,6       12 3412352 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.fdx
(deleted)
java    20742 xxx  428r   REG                8,6       10 3412365 /home/xxx/Desktop/mp.home/dev/index/Artist/_fb.nrm
(deleted)
java    20742 xxx  429r   REG                8,6       21 3412357 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.frq
java    20742 xxx  432r   REG                8,6       21 3412358 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.prx
java    20742 xxx  433r   REG                8,6       61 3412347 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.fdt
java    20742 xxx  434r   REG                8,6       28 3412348 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.fdx
java    20742 xxx  445r   REG                8,6       22 3412360 /home/xxx/Desktop/mp.home/dev/index/Artist/_fp.nrm
{code}

The application reaches the limit of maximum number of open files and then stops. 

Are we doing anything wrong here or does the bug still exist in version 3.0.3? 
Any advice is welcome.
Thanks!

> Don't leak deleted open file handles with pooled readers
> --------------------------------------------------------
>
>                 Key: LUCENE-2762
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2762
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.9.4, 3.0.3, 3.1, 4.0
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
>         Attachments: LUCENE-2762.patch
>
>
> If you have CFS enabled today, and pooling is enabled (either directly
> or because you've pulled an NRT reader), IndexWriter will hold open
> SegmentReaders against the non-CFS format of each merged segment.
> So even if you close all NRT readers you've pulled from the writer,
> you'll still see file handles open against files that have been
> deleted.
> This count will not grow unbounded, since it's limited by the number
> of segments in the index, but it's still a serious problem since the
> app had turned off CFS in the first place presumably to avoid risk of
> too-many-open-files.  It's also bad because it ties up disk space
> since these files would otherwise be deleted.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message