lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <karl.wri...@nokia.com>
Subject Trunk file handle leak?
Date Fri, 10 Sep 2010 17:27:39 GMT
Hi folks,

I am running into what appears to be a file handle leak in trunk during indexing.  It's not
clear yet what the causative event is, although the indexing runs for more than an hour before
it occurs.  The system is Ubuntu, and has 1024 file handles (per process).  This is on a trunk
checkout from about 3 hours ago.

The exception I start seeing is:

     [java] Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException:
java.io.FileNotFoundException: /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx
(Too many open files)
     [java]     at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:471)
     [java]     at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:435)
     [java] Caused by: java.io.FileNotFoundException: /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx
(Too many open files)
     [java]     at java.io.RandomAccessFile.open(Native Method)
     [java]     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
     [java]     at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:69)
     [java]     at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:90)
     [java]     at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:91)
     [java]     at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
     [java]     at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:104)
     [java]     at org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:243)
     [java]     at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:538)
     [java]     at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:635)
     [java]     at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3976)
     [java]     at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3628)
     [java]     at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:339)
     [java]     at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:407)


Here's the ulimit -a output:

root@duck6:~/solr-dym/solr-dym# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The actual number of files in the index at this time is relatively low:

root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# ls -1 | wc
    179     179    1532
root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index#

Anyone willing to work with me to narrow down the problem?

Thanks,
Karl



Mime
View raw message