couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Multiple couchjs processes being spawned
Date Sun, 02 Jan 2011 21:53:18 GMT
These reports do sound suspiciously like the problems described in COUCHDB-901 that caused
us to rewrite the OS process management in BigCouch.  Mahesh, yours is the first report I've
heard that specifically implicated view compaction.  That's significant.  I didn't understand
what you meant by "clobber all the views", though.

I'll poke around and see if I can devise a scenario where the OS process would not be released
after view compaction.  In my experience the internal ets tables maintained in CouchDB to
track OS processes got out-of-sync - one table would report 1000 available processes, while
another table would claim only 2.  Best,

Adam

On Jan 2, 2011, at 12:25 PM, Mahesh Paolini-Subramanya wrote:

> Sorry - apologies - hit Send by accident :-(
> 
> Wow - wierdness here.  I'm running 1.0.1, and have noticed the same thing
> happening.  The specific (and very replicable - for me at least) scenario is
> as follows
> - I've got around a thousand dbs, each with around 50K documents, and each
> with about 5 (javascript) design documents.
> - I clobber all the views, and remotely (LWP::UserAgent) compact the view
> ($db/_compact/$design)
> - THe compaction of the design document results in the design exactly being
> generate (this is good_
> - The compaction also leave a couchjs process just lying around (this would
> be bad)
> - After a few thousand couchjs processes, couchdb just chokes.
> 
> Any ideas? This related in any way to the process issues in BigCouch (Jira
> -901)?
> 
> cheers
> 
> 
> 
> 
> On Mon, Sep 7, 2009 at 12:18 AM, Paul Davis <paul.joseph.davis@gmail.com>wrote:
> 
>> Another random thought, after your clients have been running a bit, an
>> easy way to check why type of stuff is going on with sockets is:
>> 
>> $ netstat -ap tcp
>> 
>> If your apps have periods of high turnover, check the sockets to see
>> if you have lots of them in TIME_WAIT state.
>> 
>> Paul
>> 
>> On Sun, Sep 6, 2009 at 11:39 PM, Arun Thampi<arun.thampi@gmail.com> wrote:
>>> Thanks Paul for the quick response. I'll poke around further to see if
>> there
>>> is anything amiss and will keep the list posted.
>>> Cheers,
>>> Arun
>>> 
>>> On Mon, Sep 7, 2009 at 11:24 AM, Paul Davis <paul.joseph.davis@gmail.com
>>> wrote:
>>> 
>>>> The algorithm for CouchDB's use of OS processes is pretty simple:
>>>> 
>>>> "Give me an os process plz"
>>>> 
>>>> If none are available, then it creates a new one. A large number of OS
>>>> processes would suggest to me that something isn't releasing OS
>>>> processes correctly. My first guess would be to look at _list and
>>>> _show as those both require an OS process, but are also at the mercy
>>>> of client connection semantics. I might be reaching a bit, but I
>>>> wonder if CouchRest is failing to close sockets properly. Its not much
>>>> more than a random finger pointing, but there have been other errors
>>>> recently that also suggest CouchRest isn't doing proper socket
>>>> handling.
>>>> 
>>>> The other possibility is there's something weird with killing
>>>> processes, but couchspawnkillable should've fixed that.
>>>> 
>>>> HTH,
>>>> Paul Davis
>>>> 
>>>> On Sun, Sep 6, 2009 at 11:18 PM, Arun Thampi<arun.thampi@gmail.com>
>> wrote:
>>>>> Hi guys - Been running CouchDB trunk(r804727) in production for about
>> 3
>>>>> weeks now and one thing I've noticed is that the number of couchjs
>>>> processes
>>>>> (/usr/local/lib/couchdb/bin/couchjs
>>>> /usr/local/share/couchdb/server/main.js)
>>>>> keeps increasing to a large amount. Is this normal? Does CouchDB
>> manage
>>>>> these processes and eventually kill inactive couchjs processes?
>>>>> Just FYI I'm using CouchRest as part of a Rails app to query two
>>>> different
>>>>> views in my db.
>>>>> 
>>>>> Thanks in advance.
>>>>> 
>>>>> Cheers,
>>>>> Arun
>>>>> 
>>>>> --
>>>>> It's better to be a pirate than join the Navy - Steve Jobs
>>>>> http://mclov.in
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> It's better to be a pirate than join the Navy - Steve Jobs
>>> http://mclov.in
>>> 
>> 
> 
> 
> 
> -- 
> Mahesh Paolini-Subramanya | CTO | mahesh@aptela.com | 703.386.1500 Ext. 9100
> 13454 Sunrise Valley Drive | Suite 500 | Herndon, VA


Mime
View raw message