couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eric casteleijn <eric.castele...@canonical.com>
Subject Running out of erlang processes.
Date Tue, 10 Nov 2009 19:10:07 GMT
Today our production couchdb suddenly crashed and refused to restart, 
(scrubbed end of the logfile included.) After asking on #couchdb, Jan 
kindly helped me understand that this was due to running into the 
default limit of allowed erlang processes (32K and a bit.) and suggested 
increasing the number.

While I have no problem doing that if it solves the problem, I am still 
left with some questions that I hope you all can help me answer.

I'll first sketch the set-up of our system very globabally, and tell you 
what we were doing at the time that might have had an impact:

We have a single server node, with a lot of clients replicating to and 
from databases it contains, and a web interface reading from and writing 
to it. The server node contains somewhere between 200K and 300K databases.

I was rather naively doing collection some stats from the server by 
running something like the following pseudo code:

all_dbs = GET "/_all_dbs"
for db in all_dbs:
     sleep for a little bit
     GET "/" + db

to get the number of databases and the total number of documents in all 
databases. After some minutes of running this, the crash occured. This 
might not be related to us running the script, but that would be one 
hell of a coincedence, as we haven't seen this particular error before. 
Also after aborting the script, couchdb seemed to slowly recuperate.

My questions in declining order of ulcerinducingness:

1. Can anyone explain why the above would cause CouchDB to run out of 
processes?

2. Are there more scenarios like this that can cause such crashes?

3. Is there a way to monitor the number of processes in use? (Ideally I 
would like to have it in _stats, although I don't know how feasible that 
is.)

4. If this is not an outright bug in CouchDB, is there any way to 
degrade a little more gracefully in cases like this? It reminds me of 
the errors CouchDB throws when running out of file descriptors. I would 
things to slow down and maybe time out on connections, rather than cause 
crashes, or (in the case of file descriptors) return server errors.

-- 
- eric casteleijn
https://launchpad.net/~thisfred
http://www.canonical.com

Mime
View raw message