lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <karl.wri...@nokia.com>
Subject RE: Trunk file handle leak?
Date Fri, 10 Sep 2010 17:51:19 GMT
Hi Simon,

(1) There are periodic commits, every 10,000 records.
(2) I have no searcher/reader open at the same time, that I am aware of.  This is a straight
indexing task.  (You ought to know, you wrote some of the code!)
(3) I *do* see auto warming being called, but it seems not to happen at the same time as a
commit, but rather afterwards.

Once it starts happening, this happens repeatedly on every commit.

Karl


-----Original Message-----
From: ext Simon Willnauer [mailto:simon.willnauer@googlemail.com] 
Sent: Friday, September 10, 2010 1:46 PM
To: dev@lucene.apache.org
Subject: Re: Trunk file handle leak?

Karl,

this does not seem to be a leaking file handle problem. What you are
seeing when you do a ls -1 | wc is you index after the failed merge. A
single merge exception does not mean that all concurrently running
merges failed. There might have been more than 1024 open files during
that time but just a single merge thread hit the limit. It really
depends on you IW settings, ram buffer etc. Especially if you have a
searcher / reader open you OS will keep the files open while you are
not seeing them anymore with ls -1 Those files are kept open until the
searcher is closed. Are you indexing with intermediate commits or do
you do any kind of batches with fixed size and subsequently warming
the searcher?

Can you try to raise the limit of open files?

simon

On Fri, Sep 10, 2010 at 7:27 PM,  <karl.wright@nokia.com> wrote:
> Hi folks,
>
> I am running into what appears to be a file handle leak in trunk during
> indexing.  It’s not clear yet what the causative event is, although the
> indexing runs for more than an hour before it occurs.  The system is Ubuntu,
> and has 1024 file handles (per process).  This is on a trunk checkout from
> about 3 hours ago.
>
> The exception I start seeing is:
>
>      [java] Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.io.FileNotFoundException:
> /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many open
> files)
>      [java]     at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:471)
>      [java]     at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:435)
>      [java] Caused by: java.io.FileNotFoundException:
> /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many open
> files)
>      [java]     at java.io.RandomAccessFile.open(Native Method)
>      [java]     at
> java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>      [java]     at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:69)
>      [java]     at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:90)
>      [java]     at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:91)
>      [java]     at
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
>      [java]     at
> org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:104)
>      [java]     at
> org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:243)
>      [java]     at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:538)
>      [java]     at
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:635)
>      [java]     at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3976)
>      [java]     at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3628)
>      [java]     at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:339)
>      [java]     at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:407)
>
>
> Here’s the ulimit –a output:
>
> root@duck6:~/solr-dym/solr-dym# ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 20
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 16382
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) unlimited
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> The actual number of files in the index at this time is relatively low:
>
> root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# ls -1 | wc
>     179     179    1532
> root@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index#
>
> Anyone willing to work with me to narrow down the problem?
>
> Thanks,
> Karl
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Mime
View raw message