lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Sutton <adr...@symphonious.net>
Subject Re: DirectSolrConnection, write.lock and Too Many Open Files
Date Mon, 10 Sep 2007 23:11:20 GMT
On 11/09/2007, at 8:46 AM, Ryan McKinley wrote:
> lucene opens a lot of files.  It can easily get beyond 1024. (I  
> think the default).  I'm no expert on how the file handling works,  
> but I think more files are open if you are searching and writing at  
> the same time.
>
> If you can't increase the limit you can try:
>  <useCompoundFile>true</useCompoundFile>
>
> It is slower, but if you are unable to change the ulimit on the  
> deployed machines....

I've done a bit of poking on the server and ulimit doesn't seem to be  
the problem:
e2wiki:~$ ulimit
unlimited
e2wiki:~$ cat /proc/sys/fs/file-max
170355

So there's either something going on behind my back (quite possible,  
it's a VM) or lucene is opening a really insane number of files. I  
did check that those values were the same for the tomcat55 user that  
Tomcat actually runs as.  An lsof -p on the Tomcat process always  
shows 40 files in use, the total open files sits around 1000-1500  
even when reindexing all the content. I'll watch it a bit more over  
time and see what happens.

I notice that Confluence recommends at least 200000 for the max file  
limit, at least before they switched to compound indexing so it's  
possible that the 170355 limit could be reached, but it seems  
unlikely with our load.

> If you need to use this in production soon, I'd suggest sticking  
> with 1.2 for a while.  There has been a LOT of action in trunk and  
> it may be good to let it settle before upgrading a production system.
>
> You should not need to upgrade to fix the write.lock and Too Many  
> Open Files problem.  Try increasing ulimit or using a compoundfile  
> before upgrading.

We're quite a way off of real production, it's just internal use at  
the moment (on the real product server, but we're a small company so  
we can handle having some problems). I'll try out the current nightly  
build and see how it goes, as much as anything out of interest but  
probably won't pull new builds very often.

Thanks again,

Adrian Sutton
http://www.symphonious.net

Mime
View raw message