lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene 2.9.0 leaves too many .cfs files open, causing too many files open java error.
Date Sun, 18 Oct 2009 20:34:44 GMT
Super!  Thanks for bringing closure.

Mike

On Sun, Oct 18, 2009 at 2:31 PM, GlenAbbeyDrive <wende.long@gmail.com> wrote:
>
> Using a 'realtime' reader off the IndexWriter (writer.getReader()) instead of
> indexReader.reopen() seems fixed this problem.
>
>
> Michael McCandless-2 wrote:
>>
>> Hmm, not good.  Can you share more details about how your app is using
>> Lucene?
>>
>> Do you also have an IndexReader that's open on this directory?  Do you
>> reopen it after indexing documents?  If so, using IndexReader.reopen
>> or by closing and opening a new IndexReader?  Or, by getting a near
>> real-time reader?
>>
>> Are you closing the IndexWriter after each batch, or calling commit?
>>
>> Mike
>>
>> On Sun, Oct 18, 2009 at 11:14 AM, GlenAbbeyDrive <wende.long@gmail.com>
>> wrote:
>>>
>>> I switched from Lucene 2.4.0 to the latest 2.9.0 version and got too many
>>> files open within a few hours from my indexing process.  Our indexing
>>> Java
>>> process adds about 2000 documents/minute.
>>> The IndexWriter (iw) has the following settings:
>>>
>>>            iw.setMaxFieldLength(1024*1024*1024); // 1G as maximum terms.
>>>            iw.setMergeFactor(10);
>>>            iw.setMaxMergeDocs(100000);
>>>            iw.setMaxBufferedDocs(1000);
>>>            iw.setRAMBufferSizeMB(128);
>>>            iw.setUseCompoundFile(true);
>>>
>>>
>>>
>>> The 'lsof' result shows rapidly growing number of open files in 'deleted'
>>> state. Counting the number of these:
>>>
>>> [server-1]#date
>>> Sun Oct 18 11:02:37 EDT 2009
>>> [server-1]# lsof -p 19606 |grep deleted  |wc -l
>>> 3187
>>> [server-1]# date
>>> Sun Oct 18 11:07:29 EDT 2009
>>> [server-1]# lsof -p 19606 |grep deleted  |wc -l
>>> 3912
>>>
>>> Here is a sample of those open files left behind in 'deleted' state:
>>>
>>> java    19606 root 1861r   REG       8,17    273561 368673054
>>> /archive/index/_5fq.cfs (deleted)
>>> java    19606 root 1862r   REG       8,17      9428 368673053
>>> /archive/index/_5fp.cfs (deleted)
>>> java    19606 root 1863r   REG       8,17    363690 368673052
>>> /archive/index/_5fo.cfs (deleted)
>>> java    19606 root 1864r   REG       8,17     47732 368673051
>>> /archive/index/_5fn.cfs (deleted)
>>> java    19606 root 1865r   REG       8,17    417453 368673050
>>> /archive/index/_5fm.cfs (deleted)
>>> java    19606 root 1866r   REG       8,17   1518126 368673049
>>> /archive/index/_5fl.cfs (deleted)
>>> java    19606 root 1871r   REG       8,17   1236023 368673057
>>> /archive/index/_5ft.cfs (deleted)
>>> java    19606 root 1872r   REG       8,17     29743 368673055
>>> /archive/index/_5fr.cfs (deleted)
>>> java    19606 root 1873r   REG       8,17    273561 368673054
>>> /archive/index/_5fq.cfs (deleted)
>>> java    19606 root 1874r   REG       8,17      9428 368673053
>>> /archive/index/_5fp.cfs (deleted)
>>> java    19606 root 1875r   REG       8,17    363690 368673052
>>> /archive/index/_5fo.cfs (deleted)
>>> java    19606 root 1876r   REG       8,17     47732 368673051
>>> /archive/index/_5fn.cfs (deleted)
>>> java    19606 root 1877r   REG       8,17    417453 368673050
>>> /archive/index/_5fm.cfs (deleted)
>>> java    19606 root 1878r   REG       8,17   1518126 368673049
>>> /archive/index/_5fl.cfs (deleted)
>>> java    19606 root 1883r   REG       8,17   1389856 368673056
>>> /archive/index/_5fs.cfs (deleted)
>>> java    19606 root 1884r   REG       8,17     29743 368673055
>>> /archive/index/_5fr.cfs (deleted)
>>> java    19606 root 1885r   REG       8,17    273561 368673054
>>> /archive/index/_5fq.cfs (deleted)
>>> java    19606 root 1886r   REG       8,17      9428 368673053
>>> /archive/index/_5fp.cfs (deleted)
>>> java    19606 root 1887r   REG       8,17    363690 368673052
>>> /archive/index/_5fo.cfs (deleted)
>>> java    19606 root 1888r   REG       8,17     47732 368673051
>>> /archive/index/_5fn.cfs (deleted)
>>> java    19606 root 1889r   REG       8,17    417453 368673050
>>> /archive/index/_5fm.cfs (deleted)
>>> java    19606 root 1890r   REG       8,17   1518126 368673049
>>> /archive/index/_5fl.cfs (deleted)
>>> java    19606 root 1895r   REG       8,17   1389856 368673056
>>> /archive/index/_5fs.cfs (deleted)
>>> java    19606 root 1896r   REG       8,17     29743 368673055
>>> /archive/index/_5fr.cfs (deleted)
>>> java    19606 root 1897r   REG       8,17    273561 368673054
>>> /archive/index/_5fq.cfs (deleted)
>>> java    19606 root 1898r   REG       8,17      9428 368673053
>>> /archive/index/_5fp.cfs (deleted)
>>> java    19606 root 1899r   REG       8,17    363690 368673052
>>> /archive/index/_5fo.cfs (deleted)
>>> java    19606 root 1900r   REG       8,17     47732 368673051
>>> /archive/index/_5fn.cfs (deleted)
>>> java    19606 root 1901r   REG       8,17    417453 368673050
>>> /archive/index/_5fm.cfs (deleted)
>>> java    19606 root 1902r   REG       8,17   1518126 368673049
>>> /archive/index/_5fl.cfs (deleted)
>>> java    19606 root 1910r   REG       8,17    403610 368673058
>>> /archive/index/_5fu.cfs (deleted)
>>> java    19606 root 1911r   REG       8,17   1236023 368673057
>>> /archive/index/_5ft.cfs (deleted)
>>> java    19606 root 1912r   REG       8,17   1389856 368673056
>>> /archive/index/_5fs.cfs (deleted)
>>> java    19606 root 1913r   REG       8,17     29743 368673055
>>> /archive/index/_5fr.cfs (deleted)
>>> java    19606 root 1914r   REG       8,17    273561 368673054
>>> /archive/index/_5fq.cfs (deleted)
>>> java    19606 root 1915r   REG       8,17      9428 368673053
>>> /archive/index/_5fp.cfs (deleted)
>>> java    19606 root 1916r   REG       8,17    363690 368673052
>>> /archive/index/_5fo.cfs (deleted)
>>> java    19606 root 1917r   REG       8,17     47732 368673051
>>> /archive/index/_5fn.cfs (deleted)
>>> java    19606 root 1918r   REG       8,17    417453 368673050
>>> /archive/index/_5fm.cfs (deleted)
>>> java    19606 root 1919r   REG       8,17   1518126 368673049
>>> /archive/index/_5fl.cfs (deleted)
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25947462.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25949130.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message