couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hingston <mark.hings...@vardr.com>
Subject Couchdb problem near period of low disk space
Date Fri, 19 Oct 2012 04:48:48 GMT
Hi guys,

Apologies if this has been posted already but I couldn't find any 
related topics in the archives. I saw what looked like a crash in my 
couchdb server the other day - the relevant log section is included 
below. I'm running couch version 1.2.0 on an Amazon EC2 instance running 
Ubuntu 12.04.1.

It took about a day for this issue to get noticed and during that time 
the server basically ran out of disk space:

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      8.0G  7.6G     0 100% /

And creating small files is possible but creating large ones fails. It 
seems possible that the disk filled around the time of the the couch 
crash because the end of the log file does seem to be truncated (the 
last thing I see is the "{summary, [" and "<<128,0,0,154>>," lines which 
seem to have been cut off before the closing braces).

Also after the crash happened couch didn't seem to actually shutdown, as 
I could still see the couchdb erlang process running and couch seemed to 
be stuck in a state where no clients could connect to it. After I 
stopped couch and cleared off the drive and then restarted the machine 
clients could connect to it again and all seemed fine.

However, it's not obvious to me from the log / shutdown report that this 
was related to disk space at all. Should I expect couch to log a warning 
in a low disk space situation? Should this situation be recoverable? Can 
anyone from the list help shed some light on what happened here?

Thanks,
Mark


[Mon, 15 Oct 2012 07:39:50 GMT] [info] [<0.22545.20>] 10.123.89.44 - - 
GET /jungle/264529?conflicts=true 200
[Mon, 15 Oct 2012 07:39:50 GMT] [error] [<0.174.0>] ** Generic server 
<0.174.0> terminating
** Last message in was {update_docs,<0.23330.20>,
                            [[{{doc,<<"2709-31ff">>,
                                   {1,
[<<222,44,166,107,227,110,24,30,56,61,153,
                                       116,242,241,242,189>>]},
                                   {summary,
                                       [<<128,0,0,154>>,
<<120,60,65,52,122,142,23,21,233,172,79,
                                          5,217,11,12,237>>,
<<131,104,2,109,0,0,0,139,131,104,1,
108,0,0,0,4,104,2,109,0,0,0,9,116,
                               [Mon, 15 Oct 2012 07:39:51 GMT] [error] 
[<0.23326.20>] Uncaught server error: {kill,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23328.20>] Uncaught server 
error: {kill,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22409.20>] Uncaught server 
error: {kill,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22545.20>] Uncaught server 
error: {kill,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22410.20>] Uncaught server 
error: {kill,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23325.20>] Uncaught server 
error: {noproc,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.20484.20>] Uncaught server 
error: {noproc,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22547.20>] Uncaught server 
error: {noproc,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22378.20>] Uncaught server 
error: {noproc,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23251.20>] Uncaught server 
error: {noproc,
                                                {gen_server,call,
                                                 [couch_server,
{open,<<"jungle">>,
                                                   [{user_ctx,
                                                     {user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
                                                  infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23326.20>] 10.123.89.44 - - 
GET /jungle/7452545?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23328.20>] 10.123.89.44 - - 
GET /jungle/1883403?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22409.20>] 10.123.89.44 - - 
GET /jungle/7508765?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22545.20>] 10.123.89.44 - - 
GET /jungle/3713848?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22410.20>] 10.123.89.44 - - 
GET /jungle/5153254?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23325.20>] 10.123.89.44 - - 
GET /jungle/4742903?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.20484.20>] 10.123.89.44 - - 
GET /jungle/5890907?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22547.20>] 10.123.89.44 - - 
GET /jungle/9617720?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22378.20>] 10.123.89.44 - - 
GET /jungle/912831?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23251.20>] 10.123.89.44 - - 
GET /jungle/4130419?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23967.20>] ** Generic server 
<0.23967.20> terminating
** Last message in was {update_docs,<0.23333.20>,
                            [[{{doc,<<"2709-31ff">>,
                                   {1,
[<<222,44,166,107,227,110,24,30,56,61,153,
                                       116,242,241,242,189>>]},
                                   {summary,
                                       [<<128,0,0,154>>,

Mime
View raw message