lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uwe Schindler <...@thetaphi.de>
Subject Re: “Too many open files" error hit when I run my Lucene 3.0.3 application on Java 8
Date Wed, 22 Feb 2017 16:49:31 GMT
The garbage collector in Java 8 may have different behavior. This may cause issues if you have
buggy code that does not close indexreaders open in parallel to indexing. If it relies on
Garbage Collector to close accidentally open files, that won't work anymore on Java 8. The
reason for this might be changes to the finalists of Random access files in this issue for
Java 7: https://bugs.openjdk.java.net/browse/JDK-6322678

Please make sure that you close all IndexWriters and IndexReaders after use, so cleanup does
not rely on GC. 

I am also not sure if there are bugs in this old Lucene version that might fail to close all
streams correctly. This may cause these issues in combination with the above fixes in Java
8.

Uwe

Am 22. Februar 2017 17:27:48 MEZ schrieb Leonid Bolshinsky <leonidbol@gmail.com>:
>No code changes between running in Java 6 and Java 8. The same code
>still
>runs successfully with Java 6 while is failing with Java 8.
>
>On Feb 22, 2017 5:52 PM, "Frederik Van Hoyweghen" <
>frederik.vanhoyweghen@chapoo.com> wrote:
>
>Did you make any changes to your code between running on java 6 and 8?
>The java file API has changed considerably since java 8.
>
>If you -have- rewritten some of the file handling code in your indexing
>process, make sure to explicitly close the streams you create, or use
>the
>(since java 7) try-with-resources construct.
>
>
>On 22/02/2017 16:18, Leonid Bolshinsky wrote:
>
>> I have a search engine based on Lucene 3.0.3 and I can't change the
>Lucene
>> version for reasons that are out of scope of this question. Now I
>have a
>> requirement to move from Java 6 to Java 8, however when I run the
>indexing
>> using Java 8 JVM, I hit "Too many open files issue" as below:
>>
>> java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open
>files)
>>      at java.io.RandomAccessFile.open0(Native Method)
>>      at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
>>      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
>>      at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutpu
>> t.<init>(SimpleFSDirectory.java:180)
>>      at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDir
>> ectory.java:85)
>>      at
>org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
>>      at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(
>> StoredFieldsWriter.java:66)
>>      at org.apache.lucene.index.StoredFieldsWriter.finishDocument(St
>> oredFieldsWriter.java:144)
>>      at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(
>> StoredFieldsWriter.java:192)
>>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocum
>> ent(DocumentsWriter.java:1559)
>>      at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(Docume
>> ntsWriter.java:1578)
>>      at org.apache.lucene.index.DocumentsWriter.finishDocument(Docum
>> entsWriter.java:1155)
>>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
>> entsWriter.java:832)
>>      at org.apache.lucene.index.DocumentsWriter.updateDocument(Docum
>> entsWriter.java:807)
>>      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWrit
>> er.java:2155)
>>
>> The ulimit on my system is 80.000 files and when I run lsof, I get
>that the
>> open files belong to the index and they are marked as (deleted)
>segments. I
>> am aware that there are tons of questions, answers and blogs about
>the
>> Lucene open files issue, however:
>>
>>     - I've already tried whatever is recommended (review the code
>looking
>>
>>     for open IndexWriters, increase ulimit, enable compound mode,
>decrease
>> the
>>     number of segments from 10 to 4, review the code for open files)
>> without
>>     success and
>>     - The issue didn't appear with Java 6 so I am guessing that the
>problem
>>
>>     is in how Java 8 handles files.
>>
>> Anybody faced this issue with Java 8 before? Any additional idea
>about how
>> to further troubleshoot this issue or what could be the cause?
>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message