Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Date: Wed, 22 Feb 2017 16:49:31 +0000
User-Agent: K-9 Mail for Android
In-Reply-To: <CAL5+m1BPLd6Qqaesww_mr0=CO9bxP5basXodP91-=nwmB6fbtg@mail.gmail.com>
References: <CAL5+m1DZFZP_VTXwxJZHoQQ3ia8Jspgp1Yr_4GToZFH_feV=jw@mail.gmail.com> <f0733900-bd97-58ec-cb25-f890e1053f31@chapoo.com> <CAL5+m1BPLd6Qqaesww_mr0=CO9bxP5basXodP91-=nwmB6fbtg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----KG7LSQVAWIOKGC1L21VLT2MYDRSMW4"
Content-Transfer-Encoding: 7bit
Subject: =?UTF-8?Q?Re=3A_=E2=80=9CToo_many_open_files=22_error_hit_when_I?= =?UTF-8?Q?_run_my_Lucene_3=2E0=2E3_application_on_Java_8?=
To: java-user@lucene.apache.org
From: Uwe Schindler <uwe@thetaphi.de>
Message-ID: <8FA53545-D72D-4C91-804F-A5489C8C06CF@thetaphi.de>
archived-at: Wed, 22 Feb 2017 16:49:57 -0000

------KG7LSQVAWIOKGC1L21VLT2MYDRSMW4
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable

The garbage collector in Java 8 may have different behavior=2E This may cau=
se issues if you have buggy code that does not close indexreaders open in p=
arallel to indexing=2E If it relies on Garbage Collector to close accidenta=
lly open files, that won't work anymore on Java 8=2E The reason for this mi=
ght be changes to the finalists of Random access files in this issue for Ja=
va 7: https://bugs=2Eopenjdk=2Ejava=2Enet/browse/JDK-6322678

Please make sure that you close all IndexWriters and IndexReaders after us=
e, so cleanup does not rely on GC=2E=20

I am also not sure if there are bugs in this old Lucene version that might=
 fail to close all streams correctly=2E This may cause these issues in comb=
ination with the above fixes in Java 8=2E

Uwe

Am 22=2E Februar 2017 17:27:48 MEZ schrieb Leonid Bolshinsky <leonidbol@gm=
ail=2Ecom>:
>No code changes between running in Java 6 and Java 8=2E The same code
>still
>runs successfully with Java 6 while is failing with Java 8=2E
>
>On Feb 22, 2017 5:52 PM, "Frederik Van Hoyweghen" <
>frederik=2Evanhoyweghen@chapoo=2Ecom> wrote:
>
>Did you make any changes to your code between running on java 6 and 8?
>The java file API has changed considerably since java 8=2E
>
>If you -have- rewritten some of the file handling code in your indexing
>process, make sure to explicitly close the streams you create, or use
>the
>(since java 7) try-with-resources construct=2E
>
>
>On 22/02/2017 16:18, Leonid Bolshinsky wrote:
>
>> I have a search engine based on Lucene 3=2E0=2E3 and I can't change the
>Lucene
>> version for reasons that are out of scope of this question=2E Now I
>have a
>> requirement to move from Java 6 to Java 8, however when I run the
>indexing
>> using Java 8 JVM, I hit "Too many open files issue" as below:
>>
>> java=2Eio=2EFileNotFoundException: /myIndex/_27c=2Efdx (Too many open
>files)
>>      at java=2Eio=2ERandomAccessFile=2Eopen0(Native Method)
>>      at java=2Eio=2ERandomAccessFile=2Eopen(RandomAccessFile=2Ejava:333=
)
>>      at java=2Eio=2ERandomAccessFile=2E<init>(RandomAccessFile=2Ejava:2=
57)
>>      at org=2Eapache=2Elucene=2Estore=2ESimpleFSDirectory$SimpleFSIndex=
Outpu
>> t=2E<init>(SimpleFSDirectory=2Ejava:180)
>>      at org=2Eapache=2Elucene=2Estore=2ENIOFSDirectory=2EcreateOutput(N=
IOFSDir
>> ectory=2Ejava:85)
>>      at
>org=2Eapache=2Elucene=2Eindex=2EFieldsWriter=2E<init>(FieldsWriter=2Ejava=
:86)
>>      at org=2Eapache=2Elucene=2Eindex=2EStoredFieldsWriter=2EinitFields=
Writer(
>> StoredFieldsWriter=2Ejava:66)
>>      at org=2Eapache=2Elucene=2Eindex=2EStoredFieldsWriter=2EfinishDocu=
ment(St
>> oredFieldsWriter=2Ejava:144)
>>      at org=2Eapache=2Elucene=2Eindex=2EStoredFieldsWriter$PerDoc=2Efin=
ish(
>> StoredFieldsWriter=2Ejava:192)
>>      at org=2Eapache=2Elucene=2Eindex=2EDocumentsWriter$WaitQueue=2Ewri=
teDocum
>> ent(DocumentsWriter=2Ejava:1559)
>>      at org=2Eapache=2Elucene=2Eindex=2EDocumentsWriter$WaitQueue=2Eadd=
(Docume
>> ntsWriter=2Ejava:1578)
>>      at org=2Eapache=2Elucene=2Eindex=2EDocumentsWriter=2EfinishDocumen=
t(Docum
>> entsWriter=2Ejava:1155)
>>      at org=2Eapache=2Elucene=2Eindex=2EDocumentsWriter=2EupdateDocumen=
t(Docum
>> entsWriter=2Ejava:832)
>>      at org=2Eapache=2Elucene=2Eindex=2EDocumentsWriter=2EupdateDocumen=
t(Docum
>> entsWriter=2Ejava:807)
>>      at org=2Eapache=2Elucene=2Eindex=2EIndexWriter=2EupdateDocument(In=
dexWrit
>> er=2Ejava:2155)
>>
>> The ulimit on my system is 80=2E000 files and when I run lsof, I get
>that the
>> open files belong to the index and they are marked as (deleted)
>segments=2E I
>> am aware that there are tons of questions, answers and blogs about
>the
>> Lucene open files issue, however:
>>
>>     - I've already tried whatever is recommended (review the code
>looking
>>
>>     for open IndexWriters, increase ulimit, enable compound mode,
>decrease
>> the
>>     number of segments from 10 to 4, review the code for open files)
>> without
>>     success and
>>     - The issue didn't appear with Java 6 so I am guessing that the
>problem
>>
>>     is in how Java 8 handles files=2E
>>
>> Anybody faced this issue with Java 8 before? Any additional idea
>about how
>> to further troubleshoot this issue or what could be the cause?
>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene=2Eapache=2Eorg
>For additional commands, e-mail: java-user-help@lucene=2Eapache=2Eorg

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www=2Ethetaphi=2Ede
------KG7LSQVAWIOKGC1L21VLT2MYDRSMW4--