couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Goodall (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COUCHDB-1171) Multiple requests to _changes feed causes {error, system_limit} "Too many processes"
Date Thu, 26 May 2011 22:54:47 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039981#comment-13039981
] 

Matt Goodall commented on COUCHDB-1171:
---------------------------------------

I think there may be a problem here actually. What's happening is that couch_httpd_db:handle_changes_req1/2
always calls couch_stats_collector:track_process_count/1 and that spawns a MonitorFun process
to watch for the end of the _changes process. However, MonitorFun blocks on its receive until
the TCP socket is closed by the client, even when a normal or longpoll request's chunks have
been sent and the request is complete.

That means that a client that reuses HTTP connections, and happens to GET /some_db/_changes,
may eventually starve CouchDB of processes. As it happens, the next request to CouchDB also
fails which basically restarts CouchDB and clears the problem :).

It's really easy to repeat: simply add a "+P 256" to the erl command line to give erl a tiny
number of processes to play with and run the following Python code:

    import httplib
    import itertools

    conn = httplib.HTTPConnection('localhost:5984')
    for i in itertools.count():
        if not i % 100:
            print i, '...'
        conn.request('GET', '/scratch/_changes')
        resp = conn.getresponse()
        if resp.status != 200:
            break
        resp.read()

    print 'error after:', i

I can only manage 139 requests here before CouchDB runs out of processes.

The fix is probably to explicitly call couch_stats_collector's increment/1 and decrement/1
functions from the couch_httpd_db module instead of calling couch_stats_collector:track_process_count/1
and relying on the process to end.

> Multiple requests to _changes feed causes {error, system_limit} "Too many processes"
> ------------------------------------------------------------------------------------
>
>                 Key: COUCHDB-1171
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1171
>             Project: CouchDB
>          Issue Type: Bug
>    Affects Versions: 1.0.2, 1.0.3, 1.1
>            Reporter: Alexander Shorin
>
> Originally I have investigated of issue 182 of couchdb-python package where calling db.changes()
function over 32768 times generates next messages in CouchDB log:
> [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] 127.0.0.1 - - 'GET' /test/_changes
200
> [Thu, 19 May 2011 14:03:26 GMT] [error] [emulator] Too many processes
> [Thu, 19 May 2011 14:03:26 GMT] [error] [<0.2909.0>] Uncaught error in HTTP request:
{error,system_limit}
> [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] Stacktrace: [{erlang,spawn,
>                      [erlang,apply,
>                       [#Fun<couch_stats_collector.1.123391259>,[]]]},
>              {erlang,spawn,1},
>              {couch_httpd_db,handle_changes_req,2},
>              {couch_httpd_db,do_db_req,2},
>              {couch_httpd,handle_request_int,5},
>              {mochiweb_http,headers,5},
>              {proc_lib,init_p_do_apply,3}]
> [Thu, 19 May 2011 14:03:26 GMT] [info] [<0.2909.0>] 127.0.0.1 - - 'GET' /test/_changes
500
> More info about this issue could be found there: http://code.google.com/p/couchdb-python/issues/detail?id=182
> However, I still couldn't reproduce this error using only httplib module, but I've got
that same behavior using feed=longpool option:
> from httplib import HTTPConnection
> def test2():
>     conn = HTTPConnection('localhost:5984')
>     conn.connect()
>     i = 0
>     while(True):
>         conn.putrequest('GET', '/test/_changes?feed=longpool')
>         conn.endheaders()
>         conn.getresponse().read()
>         i = i + 1
>         if i % 100 == 0:
>             print i
> When i get's around 32667 exception raises
> Traceback (most recent call last):
>   File "/home/kxepal/projects/couchdb-python/issue-182/test.py", line 259, in <module>
>     test2()
>   File "/home/kxepal/projects/couchdb-python/issue-182/test.py", line 239, in test2
>     resp.read()
>   File "/usr/lib/python2.6/httplib.py", line 522, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.6/httplib.py", line 565, in _read_chunked
>     raise IncompleteRead(''.join(value))
> httplib.IncompleteRead: IncompleteRead(0 bytes read)
> [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] 127.0.0.1 - - 'GET' /test/_changes?feed=longpool
200
> [Thu, 19 May 2011 14:10:20 GMT] [error] [emulator] Too many processes
> [Thu, 19 May 2011 14:10:20 GMT] [error] [<0.3240.4>] Uncaught error in HTTP request:
{error,system_limit}
> [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] Stacktrace: [{erlang,spawn,
>                      [erlang,apply,
>                       [#Fun<couch_stats_collector.1.123391259>,[]]]},
>              {erlang,spawn,1},
>              {couch_httpd_db,handle_changes_req,2},
>              {couch_httpd_db,do_db_req,2},
>              {couch_httpd,handle_request_int,5},
>              {mochiweb_http,headers,5},
>              {proc_lib,init_p_do_apply,3}]
> [Thu, 19 May 2011 14:10:20 GMT] [info] [<0.3240.4>] 127.0.0.1 - - 'GET' /test/_changes?feed=longpool
500
> Same error. I know, that test function is quite outside from real use case, but is this
correct behavior and couldn't it be used in malicious aims?
> This exception occurres only for multiple requests within single connection for changes
feed, chunked lists or attachments are not affected, if I've done all right.
> Test environment:
> Gentoo Linux 2.6.38
> CouchDB 1.0.2 release
> couchdb-python@63feefd9e3b6
> Python 2.6.6
> If there is needed some additional information I could try to provide it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message