lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Rosenthal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (SOLR-1911) File descriptor leak while indexing, may cause index corruption
Date Mon, 17 May 2010 17:31:43 GMT

     [ https://issues.apache.org/jira/browse/SOLR-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Simon Rosenthal updated SOLR-1911:
----------------------------------

    Attachment: indexlsof.tar.gz

OK.. I built from latest trunk, used the schema associated with the index and example solrconfig.xml,
as you asked.

- Started with a snapshot of the index taken before this issue reared its head

- used post.sh to add a file with around 800 documents (different one each time)
- did a commit (no autocommit)
- did an lsof on the process 

repeated the add/commit/lsof  5 times.

The attached tarball contains the lsof outputs, and we're still seeing the number of fds incrementing
by 38-40 after each commit. I didn't go to the bitter end, but I assume we'd get there...

Here's a clue -?? I looked for file descriptors associated with one .prx file that was present
in the original snapshot in each lsof output

grep -c _r8.prx lsof.*
lsof.0:1
lsof.1:2
lsof.2:3
lsof.3:4
lsof.4:5
lsof.5:6

The .frq files seem to have the same pattern.

I'm assuming that's not good...


> File descriptor leak while indexing, may cause index corruption
> ---------------------------------------------------------------
>
>                 Key: SOLR-1911
>                 URL: https://issues.apache.org/jira/browse/SOLR-1911
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.5
>         Environment:         Ubuntu Linux, Java build 1.6.0_16-b01
>         Solr Specification Version: 3.0.0.2010.05.12.16.17.46
> 	Solr Implementation Version: 4.0-dev exported - simon - 2010-05-12 16:17:46  -- bult
from updated trunk
> 	Lucene Specification Version: 4.0-dev
> 	Lucene Implementation Version: 4.0-dev exported - 2010-05-12 16:18:26
> 	Current Time: Thu May 13 12:21:12 EDT 2010
> 	Server Start Time:Thu May 13 11:45:41 EDT 2010
>            Reporter: Simon Rosenthal
>            Priority: Critical
>         Attachments: indexlsof.tar.gz, openafteropt.txt
>
>
> While adding documents to an already existing index using this build, the number of open
file descriptors increases dramatically until the open file per-process limit is reached (1024)
, at which point there are error messages in the log to that effect. If the server is restarted
the index may be corrupt
> commits are handled by autocommit every 60 seconds or 500 documents (usually the time
limit is reached first). 
> mergeFactor is 10.
> It looks as though each time a commit takes place, the number of open files  (obtained
from " lsof -p `cat solr.pid` | egrep ' [0-9]+r ' ") increases by 40, There are several open
file descriptors associated with each file in the index.
> Rerunning the same index updates with an older Solr (built from trunk in Feb 2010) doesn't
show this problem - the number of open files fluctuates up and down as segments are created
and merged, but stays basically constant.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message