couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Fennell" <Philip.Fenn...@bbc.co.uk>
Subject CouchDB Crash during large data import...
Date Wed, 04 Feb 2009 09:50:56 GMT
Hello,

I have been working with CouchDB recently, investigating importing large
volumes of data.

I wrote an Erlang program to traverse a large directory structure;
opening XML files, extracting some text nodes and creating a small JSON
document to send to CouchDB:

{
  "_id": "10-1233327442303600",
  "_rev": "2537955117",
  "SeriesTitle": "Some Programme Title",
  "FirstTransmissionDate": "2000-01-01"
}

The program used PUT to create each document with an ID derived from the
source data and I let rip firstly on a fairly large set of data which
created 104979 documents in about 27 minutes.

Then I took the gloves-off and set it going on a much large data set.
Over a period of six hours the import was approaching 1.5 million
documents when Couch curled-up and died. The final words from Couch were
spat-out in the terminal and log which I have included at the end of
this e-mail.

One interesting point was that the
/usr/local/couchdb/var/log/couchdb/couch.log file, had grown, since
installation to 226.9MB in size. I'm not sure if that was the cause of
Couch dieing, but when I restarted couch it get spitting-out message
about the 'gen_event handler couch_log crashed'.

Would it be possible for Couch to start a new log file each day like
Tomcat does?



CouchDB Crash Report (2009-01-30)
=================================

1,487,376 PUTS to Couch over a six hour period.

Version: CouchDB 0.9.0a
Running on CentOS 5.2 in VMWare Player on Windows XP SP 2.




Terminal output
---------------

[[{doc,<<"19663916-1233331907636990">>,
                        [<<"1653613528">>],
                        {[{<<"SeriesTitle">>,<<"NodeType not
supported">>},
                          {<<"FirstTransmissionDate">>,
                           <<"NodeType not supported">>}]},
                        [],false,[]}]],
                  [new_edits]},
              infinity]}}}}

=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** gen_event handler couch_log crashed.
** Was installed in error_logger
** Last event was: {error,<0.22.0>,
                       {<0.20937.65>,
                        "** Generic server ~p terminating \n** Last
message in was ~p~n** When Server state == ~p~n** Reason for termination
== ~n** ~p~n",
                        [<0.20937.65>,
                         {update_docs,
                             [[{doc,<<"19663916-1233331907636990">>,
                                   [<<"1653613528">>],
                                   {[{<<"SeriesTitle">>,
                                      <<"NodeType not supported">>},
                                     {<<"FirstTransmissionDate">>,
                                      <<"NodeType not supported">>}]},
                                   [],false,[]}]],
                             [new_edits]},
                         {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,
                             {db_header,0,1487376,
                                 {5315851267,7274},
                                 {5315953822,{1487376,0}},
                                 {5315951830,1487376},
                                 nil,0,nil},
                             {stream,<0.20939.65>,<0.20935.65>},
                             {btree,<0.20935.65>,
                                 {5315953822,{1487376,0}},
                                 #Fun<couch_db_updater.6.128231611>,
                                 #Fun<couch_db_updater.7.15926370>,
                                 #Fun<couch_db_updater.5.103426409>,
                                 #Fun<couch_db_updater.8.88299251>},
                             {btree,<0.20935.65>,
                                 {5315951830,1487376},
                                 #Fun<couch_db_updater.9.52395017>,
                                 #Fun<couch_db_updater.10.51571716>,
                                 #Fun<couch_btree.5.112258129>,
                                 #Fun<couch_db_updater.11.72882670>},
                             {btree,<0.20935.65>,nil,
                                 #Fun<couch_btree.0.23070627>,
                                 #Fun<couch_btree.1.117278773>,
                                 #Fun<couch_btree.2.112258129>,nil},
                             1487376,<<"infax">>,
 
"/usr/local/couchdb/var/lib/couchdb/infax.couch",
                             [],
                             {[]}},
                         {{badmatch,{{error,enospc},5315956736}},
                          [{couch_btree,'-write_node/3-lc$^0/1-0-',3},
                           {couch_btree,write_node,3},
                           {couch_btree,modify_node,4},
                           {couch_btree,modify_kpnode,6},
                           {couch_btree,modify_node,4},
                           {couch_btree,modify_kpnode,6},
                           {couch_btree,modify_node,4},
                           {couch_btree,modify_kpnode,6}]}]}}
** When handler state == {<0.20938.65>,2}
** Reason == {badarg,[{io,format,
                          [<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
                           ["Fri, 30 Jan 2009 15:39:41 GMT",error,
                            <0.20937.65>,
                            "** Generic server <0.20937.65> terminating
\r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n                           [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n                                nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n                            {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
                      {couch_log,log,5},
                      {couch_log,handle_event,2},
                      {gen_event,server_update,4},
                      {gen_event,server_notify,4},
                      {gen_event,handle_msg,5},
                      {proc_lib,init_p,5}]}

=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** Generic server couch_log terminating
** Last message in was {gen_event_EXIT,couch_log,
                        {'EXIT',
                         {badarg,
                          [{io,format,
                            [<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
                             ["Fri, 30 Jan 2009 15:39:41 GMT",error,
                              <0.20937.65>,
                              "** Generic server <0.20937.65>
terminating \r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n                           [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n                                nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n                            {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
                           {couch_log,log,5},
                           {couch_log,handle_event,2},
                           {gen_event,server_update,4},
                           {gen_event,server_notify,4},
                           {gen_event,handle_msg,5},
                           {proc_lib,init_p,5}]}}}
** When Server state == {error_logger,couch_log}
** Reason for termination ==
** {'EXIT',{badarg,[{io,format,
                        [<0.20938.65>,"[~s] [~s] [~p] ~s\r~n\r~n",
                         ["Fri, 30 Jan 2009 15:39:41
GMT",error,<0.20937.65>,
                          "** Generic server <0.20937.65> terminating
\r\n** Last message in was {update_docs,\r\n
[[{doc,<<\"19663916-1233331907636990\">>,\r\n
[<<\"1653613528\">>],\r\n
{[{<<\"SeriesTitle\">>,\r\n
<<\"NodeType not supported\">>},\r\n
{<<\"FirstTransmissionDate\">>,\r\n
<<\"NodeType not supported\">>}]},\r\n
[],false,[]}]],\r\n                           [new_edits]}\r\n** When
Server state == {db,<0.20936.65>,<0.20937.65>,nil,<0.20935.65>,\r\n
{db_header,0,1487376,\r\n
{5315851267,7274},\r\n
{5315953822,{1487376,0}},\r\n
{5315951830,1487376},\r\n                                nil,0,nil},\r\n
{stream,<0.20939.65>,<0.20935.65>},\r\n
{btree,<0.20935.65>,\r\n
{5315953822,{1487376,0}},\r\n
#Fun<couch_db_updater.6.128231611>,\r\n
#Fun<couch_db_updater.7.15926370>,\r\n
#Fun<couch_db_updater.5.103426409>,\r\n
#Fun<couch_db_updater.8.88299251>},\r\n
{btree,<0.20935.65>,\r\n
{5315951830,1487376},\r\n
#Fun<couch_db_updater.9.52395017>,\r\n
#Fun<couch_db_updater.10.51571716>,\r\n
#Fun<couch_btree.5.112258129>,\r\n
#Fun<couch_db_updater.11.72882670>},\r\n
{btree,<0.20935.65>,nil,\r\n
#Fun<couch_btree.0.23070627>,\r\n
#Fun<couch_btree.1.117278773>,\r\n
#Fun<couch_btree.2.112258129>,nil},\r\n
1487376,<<\"infax\">>,\r\n
\"/usr/local/couchdb/var/lib/couchdb/infax.couch\",\r\n
[],\r\n                            {[]}}\r\n** Reason for termination ==
\r\n** {{badmatch,{{error,enospc},5315956736}},\r\n
[{couch_btree,'-write_node/3-lc$^0/1-0-',3},\r\n
{couch_btree,write_node,3},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6},\r\n     {couch_btree,modify_node,4},\r\n
{couch_btree,modify_kpnode,6}]}\r\n"]]},
                    {couch_log,log,5},
                    {couch_log,handle_event,2},
                    {gen_event,server_update,4},
                    {gen_event,server_notify,4},
                    {gen_event,handle_msg,5},
                    {proc_lib,init_p,5}]}}

=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
** Generic server couch_view terminating
** Last message in was {'EXIT',<0.20849.65>,killed}
** When Server state == {server,"/usr/local/couchdb/var/lib/couchdb"}
** Reason for termination ==
** killed

=ERROR REPORT==== 30-Jan-2009::15:39:41 ===
{mochiweb_socket_server,235,
    {child_error,
        {noproc,
            {gen_server,call,
                [couch_server,{open,<<"infax">>,[{creds,{[]}}]}]}}}}




couch.log
---------

[Fri, 30 Jan 2009 15:39:33 GMT] [error] [<0.25547.0>] ** Generic server
<0.25547.0> terminating
** Last message in was {update_docs,
                           [[{doc,<<"19662961-1233331899042990">>,
                                 [<<"2325461573">>],
                                 {[{<<"SeriesTitle">>,
                                    <<"NodeType not supported">>},
                                   {<<"FirstTransmissionDate">>,
                                    <<"NodeType not supported">>}]},
                                 [],false,[]}]],
                           [new_edits]}
** When Server state == {db,<0.25546.0>,<0.25547.0>,nil,<0.25545.0>,
                            {db_header,0,1487376,
                                {5315851267,7274},
                                {5315953822,{1487376,0}},
                                {5315951830,1487376},
                                nil,0,nil},
                            {stream,<0.25548.0>,<0.25545.0>},
                            {btree,<0.25545.0>,
                                {5315953822,{1487376,0}},
                                #Fun<couch_db_updater.6.128231611>,
                                #Fun<couch_db_updater.7.15926370>,
                                #Fun<couch_db_updater.5.103426409>,
                                #Fun<couch_db_updater.8.88299251>},
                            {btree,<0.25545.0>,
                                {5315951830,1487376},
                                #Fun<couch_db_updater.9.52395017>,
                                #Fun<couch_db_updater.10.51571716>,
                                #Fun<couch_btree.5.112258129>,
                                #Fun<couch_db_updater.11.72882670>},
                            {btree,<0.25545.0>,nil,
                                #Fun<couch_btree.0.23070627>,
                                #Fun<couch_btree.1.117278773>,
                                #Fun<couch_btree.2.112258129>,nil},
                            1487376,<<"infax">>,
 
"/usr/local/couchdb/var/lib/couchdb/infax.couch",
                            [],
                            {[]}}
** Reason for termination ==
** {{badmatch,{{error,enospc},5315956446}},
    [{couch_btree,'-write_node/3-lc$^0/1-0-',3},
     {couch_btree,write_node,3},
     {couch_btree,modify_node,4},
     {couch_btree,modify_kpnode,6},
     {couch_btree,modify_node,4},
     {couch_btree,modify_kpnode,6},
     {couch_btree,modify_node,4},
     {couch_btree,modify_kpnode,6}]}


[Fri, 30 Jan 2009 15:39:33 GMT] [error] [<0.25547.0>]
{error_report,<0.22.0>,
    {<0.25547.0>,crash_report,
     [[{pid,<0.25547.0>},
       {registered_name,[]},
       {error_info,
           {exit,
               {{badmatch,{{error,enospc},5315956446}},
                [{couch_btree,'-write_node/3-lc$^0/1-0-',3},
                 {couch_btree,write_node,3},
                 {couch_btree,modify_node,4},
                 {couch_btree,modify_kpnode,6},
                 {couch_btree,modify_node,4},
                 {couch_btree,modify_kpnode,6},
                 {couch_btree,modify_node,4},
                 {couch_btree,modif





Regards

Philip Fennell

>XML Developer (The Forge)
>
>BBC Future Media & Technology
>Media Village, 201 Wood Lane London W12 7TP
>BC4 C4, Broadcast Centre
>
>T:	0208 0085318

http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are
not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify
the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
					

Mime
View raw message