couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eric casteleijn <eric.castele...@canonical.com>
Subject Re: Big problems with writes to _config
Date Wed, 28 Oct 2009 00:11:37 GMT
eric casteleijn wrote:
> We have the following setup:
> 
> 2 near identical public facing django servers communicating with one 
> couchdb server. The couchdb server is oauth authenticated and people can 
> access it directly (well, through an apache proxy) if they have the 
> tokens to do so. New users are signed up through these django servers, 
> after which they add the user and their tokens to couchdb. (the user 
> through a POST to _users and the tokens through PUTs to _config)
> 
> We see this failing a lot, now to the point where we think it fails all 
> the time (since all those systems have separate logs not all of which we 
> have access to, this is not trivial to piece together.)
> 
> The errors the API servers get back all look like these (the lines 
> starting with '(500':
> 
> '2009-10-27 22:35:15,357 ERROR    UbuntuOne.couch: failed to add ***** = 
> 40693 to section [oauth_token_users] of local.ini:
> 
> (500, (u'timeout', u'{gen_server,call,\n            [couch_config,\n 
>          {set,"oauth_token_users","*****","40693",true}]}'))'
> 
> '2009-10-27 22:35:20,399 ERROR    UbuntuOne.couch: failed to add ***** = 
> ***** to section [oauth_token_secrets] of local.ini:
> 
> (500, (u'timeout', u'{gen_server,call,\n            [couch_config,\n 
>          {set,"oauth_token_secrets","*****",\n 
> "*****",\n                  true}]}'))'
> 
> Corresponding errors in the couchdb.log look like:

Oops, sent before pasting that in:

[Tue, 27 Oct 2009 23:29:42 GMT] [error] [<0.9591.67>] Uncaught error in 
HTTP request: {exit,
                                  {timeout,
                                   {gen_server,call,
                                    [couch_config,
                                     {set,"oauth_token_secrets",
                                      "*****",
                                      "*****",
                                      true}]}}}

[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] Stacktrace: 
[{gen_server,call,2},
              {couch_httpd_misc_handlers,handle_config_req,1},
              {couch_httpd,handle_request,5},
              {mochiweb_http,headers,5},
              {proc_lib,init_p_do_apply,3}]

[Tue, 27 Oct 2009 23:29:42 GMT] [debug] [<0.9591.67>] httpd 500 error 
response:
  {"error":"timeout","reason":"{gen_server,call,\n 
[couch_config,\n             {set,\"oauth_token_secrets\",\"*****\",\n 
                 \"*****\",\n                  true}]}"}


[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] 91.189.89.54 - - 
'PUT' /_config/oauth_token_secrets/***** 500





> My theory was that these writes to _config fail because the local.ini is 
> somehow corrupted, but I can't access that file directly (since it has 
> users' secrets) or copy it to my machine to test this theory, and 
> helping someone who is allowed to see it look for anything weird is like 
> searching for the proverbial needle in the haystack: we have lots of 
> users, and users can have multiple tokens. Add to that the fact that you 
> cannot ever delete a line from the .ini file (DELETEs against keys in 
> _config just empty the value and leave a line like 'foo = \n'!
> 
> After speaking to Jan on the channel he proposed that it may be that the 
> gen_server message inbox overflows and the gen_server times out.
> 
> Could that be, under high load, and how can we solve this? Can we 
> increase the size of this inbox, or can we possibly have multiple 
> processes handling the access? Whether it's high load or corruption or 
> something else again, right now it looks like NO new tokens can be 
> added, and hence no new users can use our system. In short: HALP!
> 
> 


-- 
- eric casteleijn
https://launchpad.net/~thisfred
http://www.canonical.com

Mime
View raw message