lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Rosenthal (JIRA)" <>
Subject [jira] Updated: (SOLR-1911) File descriptor leak while indexing, may cause index corruption
Date Mon, 17 May 2010 17:31:43 GMT


Simon Rosenthal updated SOLR-1911:

    Attachment: indexlsof.tar.gz

OK.. I built from latest trunk, used the schema associated with the index and example solrconfig.xml,
as you asked.

- Started with a snapshot of the index taken before this issue reared its head

- used to add a file with around 800 documents (different one each time)
- did a commit (no autocommit)
- did an lsof on the process 

repeated the add/commit/lsof  5 times.

The attached tarball contains the lsof outputs, and we're still seeing the number of fds incrementing
by 38-40 after each commit. I didn't go to the bitter end, but I assume we'd get there...

Here's a clue -?? I looked for file descriptors associated with one .prx file that was present
in the original snapshot in each lsof output

grep -c _r8.prx lsof.*

The .frq files seem to have the same pattern.

I'm assuming that's not good...

> File descriptor leak while indexing, may cause index corruption
> ---------------------------------------------------------------
>                 Key: SOLR-1911
>                 URL:
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.5
>         Environment:         Ubuntu Linux, Java build 1.6.0_16-b01
>         Solr Specification Version:
> 	Solr Implementation Version: 4.0-dev exported - simon - 2010-05-12 16:17:46  -- bult
from updated trunk
> 	Lucene Specification Version: 4.0-dev
> 	Lucene Implementation Version: 4.0-dev exported - 2010-05-12 16:18:26
> 	Current Time: Thu May 13 12:21:12 EDT 2010
> 	Server Start Time:Thu May 13 11:45:41 EDT 2010
>            Reporter: Simon Rosenthal
>            Priority: Critical
>         Attachments: indexlsof.tar.gz, openafteropt.txt
> While adding documents to an already existing index using this build, the number of open
file descriptors increases dramatically until the open file per-process limit is reached (1024)
, at which point there are error messages in the log to that effect. If the server is restarted
the index may be corrupt
> commits are handled by autocommit every 60 seconds or 500 documents (usually the time
limit is reached first). 
> mergeFactor is 10.
> It looks as though each time a commit takes place, the number of open files  (obtained
from " lsof -p `cat` | egrep ' [0-9]+r ' ") increases by 40, There are several open
file descriptors associated with each file in the index.
> Rerunning the same index updates with an older Solr (built from trunk in Feb 2010) doesn't
show this problem - the number of open files fluctuates up and down as segments are created
and merged, but stays basically constant.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message