incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Leddy <m...@loop.com.br>
Subject Having purge problems
Date Mon, 03 May 2010 15:56:00 GMT
Hi,

I am currently on couchdb 0.11.0 using official debian packages with
erlang 1:13.b.4-dfsg-4 and I am having problems purging old documents.

My database is constantly receiving new data and old data (more than
six weeks) is being deleted. I have been running like this for several
months and the overhead of old deleted document 'stubs' is becoming
relevant in day to day operations such as new replications, database
compaction etc.

I decided that it would be best to purge the old deleted documents
so that the database would compact better and only contain relevant
ie. recent data.

[What I would really like would be a compact that does not include
documents that match a filter function, then I could do this on each
node independently.]

Unfortunately I am encountering problems purging the documents. I wrote
a script to process all the documents via _changes and purge the old
documents but I keep hitting documents that cannot be purged.

Here is the start of my changes feed:

curl 'localhost:5984/iris/_changes?limit=5&since=0'
{"results":[
{"seq":2,"id":"_design/admin","changes":[{"rev":"1-ea95c1898a2c779d664c1d1b71a24f33"}]},
{"seq":22435808,"id":"1259540160F2016","changes":[{"rev":"2-7dcfd742f74c79286c3f3093595a83df"}],"deleted":true},
{"seq":22435809,"id":"1259540640F2016","changes":[{"rev":"2-6bd122eb9f83c0838bc9875a1b73abaf"}],"deleted":true},
{"seq":22435810,"id":"1259616780F2443","changes":[{"rev":"2-53e2311f5de7058fbfd55979816d3efc"}],"deleted":true},
{"seq":22435811,"id":"1259616784F2443","changes":[{"rev":"2-caaff4cd1290f7807c2bcfeb6edc39e0"}],"deleted":true}
],
"last_seq":22435811}

This is a compacted copy of my main production database which is
already on seq 106280009.

When i try to purge I get a badarity error:

curl -X POST 'localhost:5984/iris/_purge' -d
'{"1259540160F2016":["2-7dcfd742f74c79286c3f3093595a83df"]}'
{"error":"{{badarity,{#Fun<couch_db_updater.25.101160745>,\n
[{2,<<124,230,79,165,16,199,208,127,32,211,160,223,180,12,3,28>>},\n
{true,4185,19290621}]}},\n [{couch_key_tree,map_leafs_simple,3},\n
{couch_key_tree,map_leafs_simple,3},\n  {couch_key_tree,map_leafs,2},\n
{couch_db_updater,'-handle_call/3-fun-2-',2},\n  {lists,mapfoldl,3},\n
{couch_db_updater,handle_call,3},\n  {gen_server,handle_msg,5},\n
{proc_lib,init_p_do_apply,3}]}","reason":"{gen_server,call,\n
[<0.28323.1>,\n             {purge_docs,[{<<\"1259540160F2016\">>,\n
[{2,\n
<<125,207,215,66,247,76,121,40,108,63,48,\n
147,89,90,131,223>>}]}]}]}"}

This is what appears in my server log:

[Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28323.1>] ** Generic server
<0.28323.1> terminating 
** Last message in was {purge_docs,[{<<"1259540160F2016">>,
                                     [{2,

<<125,207,215,66,247,76,121,40,108,
                                         63,48,147,89,90,131,223>>}]}]}
** When Server state == {db,<0.28322.1>,<0.28323.1>,nil,

<<"1272901503155514">>,<0.28320.1>,<0.28324.1>,
                            {db_header,5,106014165,0,
                                {59608213341,{24516895,36227125}},
                                {59608203141,60744020},
                                {59608273696,[]},
                                0,nil,nil,1000},
                            106014165,
                            {btree,<0.28320.1>,
                                {59608213341,{24516895,36227125}},
                                #Fun<couch_db_updater.7.132302543>,
                                #Fun<couch_db_updater.8.107957134>,
                                #Fun<couch_btree.5.124754102>,
                                #Fun<couch_db_updater.9.46112288>},
                            {btree,<0.28320.1>,
                                {59608203141,60744020},
                                #Fun<couch_db_updater.10.19027664>,
                                #Fun<couch_db_updater.11.35033879>,
                                #Fun<couch_btree.5.124754102>,
                                #Fun<couch_db_updater.12.56344865>},
                            {btree,<0.28320.1>,
                                {59608273696,[]},
                                #Fun<couch_btree.0.83553141>,
                                #Fun<couch_btree.1.30790806>,
                                #Fun<couch_btree.2.124754102>,nil},
                            106014165,<<"iris">>,

"/var/lib/couchdb/0.11.0/iris.couch",[],[],nil,
                            {user_ctx,null,[],undefined},
                            nil,1000,
                            [before_header,after_header,on_file_open]}
** Reason for termination == 
** {{badarity,{#Fun<couch_db_updater.25.101160745>,
               [{2,

<<124,230,79,165,16,199,208,127,32,211,160,223,180,12,3,28>>},
                {true,4185,19290621}]}},
    [{couch_key_tree,map_leafs_simple,3},
     {couch_key_tree,map_leafs_simple,3},
     {couch_key_tree,map_leafs,2},
     {couch_db_updater,'-handle_call/3-fun-2-',2},
     {lists,mapfoldl,3},
     {couch_db_updater,handle_call,3},
     {gen_server,handle_msg,5},
     {proc_lib,init_p_do_apply,3}]}


[Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28323.1>]
{error_report,<0.31.0>,
    {<0.28323.1>,crash_report,
     [[{initial_call,{couch_db_updater,init,['Argument__1']}},
       {pid,<0.28323.1>},
       {registered_name,[]},
       {error_info,
           {exit,
               {{badarity,
                    {#Fun<couch_db_updater.25.101160745>,
                     [{2,

<<124,230,79,165,16,199,208,127,32,211,160,223,180,12,
                         3,28>>},
                      {true,4185,19290621}]}},
                [{couch_key_tree,map_leafs_simple,3},
                 {couch_key_tree,map_leafs_simple,3},
                 {couch_key_tree,map_leafs,2},
                 {couch_db_updater,'-handle_call/3-fun-2-',2},
                 {lists,mapfoldl,3},
                 {couch_db_updater,handle_call,3},
                 {gen_server,handle_msg,5},
                 {proc_lib,init_p_do_apply,3}]},

[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
       {ancestors,

[<0.28322.1>,couch_server,couch_primary_services,couch_server_sup,
            <0.32.0>]},
       {messages,[]},
       {links,[<0.28322.1>]},
       {dictionary,[]},
       {trap_exit,false},
       {status,running},
       {heap_size,4181},
       {stack_size,24},
       {reductions,42618}],
      []]}}

[Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28291.1>] Uncaught error in
HTTP request: {exit,
                                 {{{badarity,

{#Fun<couch_db_updater.25.101160745>,
                                     [{2,

<<124,230,79,165,16,199,208,127,32,211,
                                         160,223,180,12,3,28>>},
                                      {true,4185,19290621}]}},
                                   [{couch_key_tree,map_leafs_simple,3},
                                    {couch_key_tree,map_leafs_simple,3},
                                    {couch_key_tree,map_leafs,2},
                                    {couch_db_updater,
                                     '-handle_call/3-fun-2-',2},
                                    {lists,mapfoldl,3},
                                    {couch_db_updater,handle_call,3},
                                    {gen_server,handle_msg,5},
                                    {proc_lib,init_p_do_apply,3}]},
                                  {gen_server,call,
                                   [<0.28323.1>,
                                    {purge_docs,
                                     [{<<"1259540160F2016">>,
                                       [{2,
                                         <<125,207,215,66,247,76,121,
                                           40,108,63,48,147,89,90,131,
                                           223>>}]}]}]}}}

[Mon, 03 May 2010 15:45:03 GMT] [info] [<0.28291.1>] Stacktrace:
[{gen_server,call,2},
             {couch_httpd_db,db_req,2},
             {couch_httpd_db,do_db_req,2},
             {couch_httpd,handle_request_int,5},
             {mochiweb_http,headers,5},
             {proc_lib,init_p_do_apply,3}]

[Mon, 03 May 2010 15:45:03 GMT] [info] [<0.28291.1>] 127.0.0.1 - -
'POST' /iris/_purge 500

Any suggestions would be greatly appreciated.

Thanks,

Mike





Mime
View raw message