Return-Path: Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: (qmail 63182 invoked from network); 10 Sep 2010 17:29:43 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 10 Sep 2010 17:29:43 -0000 Received: (qmail 66789 invoked by uid 500); 10 Sep 2010 17:29:42 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 66729 invoked by uid 500); 10 Sep 2010 17:29:41 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 66722 invoked by uid 99); 10 Sep 2010 17:29:41 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Sep 2010 17:29:41 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of karl.wright@nokia.com designates 192.100.105.134 as permitted sender) Received: from [192.100.105.134] (HELO mgw-mx09.nokia.com) (192.100.105.134) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Sep 2010 17:29:16 +0000 Received: from vaebh105.NOE.Nokia.com (vaebh105.europe.nokia.com [10.160.244.31]) by mgw-mx09.nokia.com (Switch-3.3.3/Switch-3.3.3) with ESMTP id o8AHQtTc029531 for ; Fri, 10 Sep 2010 12:28:53 -0500 Received: from vaebh102.NOE.Nokia.com ([10.160.244.23]) by vaebh105.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 10 Sep 2010 20:27:50 +0300 Received: from smtp.mgd.nokia.com ([65.54.30.6]) by vaebh102.NOE.Nokia.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Fri, 10 Sep 2010 20:27:45 +0300 Received: from NOK-EUMSG-01.mgdnok.nokia.com ([65.54.30.86]) by nok-am1mhub-02.mgdnok.nokia.com ([65.54.30.6]) with mapi; Fri, 10 Sep 2010 19:27:45 +0200 From: To: Date: Fri, 10 Sep 2010 19:27:39 +0200 Subject: Trunk file handle leak? Thread-Topic: Trunk file handle leak? Thread-Index: ActRDXeMU5oHCcfRRoGRWAr+UzZHAA== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: AKxm A/j3 BAdT BRRv BhWR C0Ya DS0J DywX EHi5 Eq7A GoZA GxiE H7rW IR8h LMDP Lohr;1;ZABlAHYAQABsAHUAYwBlAG4AZQAuAGEAcABhAGMAaABlAC4AbwByAGcA;Sosha1_v1;7;{BFC22F01-0C43-4068-BA2C-355FB5633038};awBhAHIAbAAuAHcAcgBpAGcAaAB0AEAAbgBvAGsAaQBhAC4AYwBvAG0A;Fri, 10 Sep 2010 17:27:39 GMT;VAByAHUAbgBrACAAZgBpAGwAZQAgAGgAYQBuAGQAbABlACAAbABlAGEAawA/AA== x-cr-puzzleid: {BFC22F01-0C43-4068-BA2C-355FB5633038} acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CF3CE3EFBCA3564185DF065952A267C853F71C2E4DNOKEUMSG01mgd_" MIME-Version: 1.0 X-OriginalArrivalTime: 10 Sep 2010 17:27:45.0200 (UTC) FILETIME=[7B1B4300:01CB510D] X-Nokia-AV: Clean X-Virus-Checked: Checked by ClamAV on apache.org --_000_CF3CE3EFBCA3564185DF065952A267C853F71C2E4DNOKEUMSG01mgd_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi folks, I am running into what appears to be a file handle leak in trunk during ind= exing. It's not clear yet what the causative event is, although the indexi= ng runs for more than an hour before it occurs. The system is Ubuntu, and = has 1024 file handles (per process). This is on a trunk checkout from abou= t 3 hours ago. The exception I start seeing is: [java] Exception in thread "Lucene Merge Thread #0" org.apache.lucene.= index.MergePolicy$MergeException: java.io.FileNotFoundException: /root/solr= -dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many open files) [java] at org.apache.lucene.index.ConcurrentMergeScheduler.handleM= ergeException(ConcurrentMergeScheduler.java:471) [java] at org.apache.lucene.index.ConcurrentMergeScheduler$MergeTh= read.run(ConcurrentMergeScheduler.java:435) [java] Caused by: java.io.FileNotFoundException: /root/solr-dym/solr-d= ym/solr_home_v2/nose/data/index/_5l.fdx (Too many open files) [java] at java.io.RandomAccessFile.open(Native Method) [java] at java.io.RandomAccessFile.(RandomAccessFile.java:21= 2) [java] at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexI= nput$Descriptor.(SimpleFSDirectory.java:69) [java] at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexI= nput.(SimpleFSDirectory.java:90) [java] at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<= init>(NIOFSDirectory.java:91) [java] at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDi= rectory.java:78) [java] at org.apache.lucene.index.FieldsReader.(FieldsReader= .java:104) [java] at org.apache.lucene.index.SegmentReader$CoreReaders.openDo= cStores(SegmentReader.java:243) [java] at org.apache.lucene.index.SegmentReader.get(SegmentReader.= java:538) [java] at org.apache.lucene.index.IndexWriter$ReaderPool.get(Index= Writer.java:635) [java] at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWri= ter.java:3976) [java] at org.apache.lucene.index.IndexWriter.merge(IndexWriter.ja= va:3628) [java] at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge= (ConcurrentMergeScheduler.java:339) [java] at org.apache.lucene.index.ConcurrentMergeScheduler$MergeTh= read.run(ConcurrentMergeScheduler.java:407) Here's the ulimit -a output: root@duck6:~/solr-dym/solr-dym# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited The actual number of files in the index at this time is relatively low: root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# ls -1 | wc 179 179 1532 root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# Anyone willing to work with me to narrow down the problem? Thanks, Karl --_000_CF3CE3EFBCA3564185DF065952A267C853F71C2E4DNOKEUMSG01mgd_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi folks,
 
I am running into what appears to be a file handle leak in trunk durin= g indexing.  It’s not clear yet what the causative event is, alt= hough the indexing runs for more than an hour before it occurs.  The s= ystem is Ubuntu, and has 1024 file handles (per process).  This is on a trunk checkout from about 3 hours ago.
 
The exception I start seeing is:
 
     [java] Exception in thread "Lucene Merge= Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.i= o.FileNotFoundException: /root/solr-dym/solr-dym/solr_home_v2/nose/data/ind= ex/_5l.fdx (Too many open files)
     [java]     at org.apache.= lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeS= cheduler.java:471)
     [java]     at org.apache.= lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeSchedu= ler.java:435)
     [java] Caused by: java.io.FileNotFoundExcepti= on: /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many = open files)
     [java]     at java.io.Ran= domAccessFile.open(Native Method)
     [java]     at java.io.Ran= domAccessFile.<init>(RandomAccessFile.java:212)
     [java]     at org.apache.= lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(S= impleFSDirectory.java:69)
     [java]     at org.apache.= lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDire= ctory.java:90)
     [java]     at org.apache.= lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.jav= a:91)
     [java]     at org.apache.= lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
     [java]     at org.apache.= lucene.index.FieldsReader.<init>(FieldsReader.java:104)
     [java]     at org.apache.= lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:243= )
     [java]     at org.apache.= lucene.index.SegmentReader.get(SegmentReader.java:538)
     [java]     at org.apache.= lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:635)
     [java]     at org.apache.= lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3976)
     [java]     at org.apache.= lucene.index.IndexWriter.merge(IndexWriter.java:3628)
     [java]     at org.apache.= lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java= :339)
     [java]     at org.apache.= lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeSchedu= ler.java:407)
 
 
Here’s the ulimit –a output:
 
root@duck6:~/solr-dym/solr-dym# ulimit -a
core file size          (= blocks, -c) 0
data seg size         &nb= sp; (kbytes, -d) unlimited
scheduling priority        &nb= sp;    (-e) 20
file size          &= nbsp;    (blocks, -f) unlimited
pending signals         &= nbsp;       (-i) 16382
max locked memory       (kbytes, -l) 64<= /div>
max memory size         (kbyte= s, -m) unlimited
open files          =             (-n) 102= 4
pipe size          &= nbsp; (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority        &nbs= p;     (-r) 0
stack size          =     (kbytes, -s) 8192
cpu time          &n= bsp;    (seconds, -t) unlimited
max user processes        &nbs= p;     (-u) unlimited
virtual memory          (= kbytes, -v) unlimited
file locks          =             (-x) unl= imited
 
The actual number of files in the index at this time is relatively low= :
 
root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# ls -1 | w= c
    179     179    1= 532
root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index#
 
Anyone willing to work with me to narrow down the problem?
 
Thanks,
Karl
 
 
 --_000_CF3CE3EFBCA3564185DF065952A267C853F71C2E4DNOKEUMSG01mgd_--