Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of jchris@gmail.com designates
 74.125.82.52 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=eSTFkSbW5kveu9U5f1ro6FJeZiO0fQlb7W/GcTS81yQhKz+jQ02n2+9eUBqP3NcYbX
         VQ4gpeEfyOYq9fuJa0y6ucjMFGik17mGVeQfoTQDZmP+o541BTJ0oKNYF0BGIGJ31TAO
         3ufiCgRn8CG41aYD61wwHLerYtFEV5Wf7ekWo=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1078)
Subject: Re: Having purge problems
From: J Chris Anderson <jchris@gmail.com>
In-Reply-To: <1272902160.30231.70.camel@mike>
Date: Tue, 4 May 2010 09:09:29 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <24B6CFE1-4663-4CA4-B142-DA80C3B27D17@gmail.com>
References: <1272902160.30231.70.camel@mike>
To: user@couchdb.apache.org


On May 3, 2010, at 8:56 AM, Mike Leddy wrote:

> Hi,
>=20
> I am currently on couchdb 0.11.0 using official debian packages with
> erlang 1:13.b.4-dfsg-4 and I am having problems purging old documents.
>=20
> My database is constantly receiving new data and old data (more than
> six weeks) is being deleted. I have been running like this for several
> months and the overhead of old deleted document 'stubs' is becoming
> relevant in day to day operations such as new replications, database
> compaction etc.
>=20
> I decided that it would be best to purge the old deleted documents
> so that the database would compact better and only contain relevant
> ie. recent data.
>=20
> [What I would really like would be a compact that does not include
> documents that match a filter function, then I could do this on each
> node independently.]
>=20
> Unfortunately I am encountering problems purging the documents. I =
wrote
> a script to process all the documents via _changes and purge the old
> documents but I keep hitting documents that cannot be purged.
>=20

Thanks for the bug report. One way I've seen people accomplish this use =
case that doesn't involve purging, is by storing documents into a new =
database each week, and then throwing out old database files.=20

Purging is really designed for removing secret data that was =
accidentally saved, more than for reclaiming space. Thanks for the bug =
report - hopefully it will be easy to fix.

Chris

> Here is the start of my changes feed:
>=20
> curl 'localhost:5984/iris/_changes?limit=3D5&since=3D0'
> {"results":[
> =
{"seq":2,"id":"_design/admin","changes":[{"rev":"1-ea95c1898a2c779d664c1d1=
b71a24f33"}]},
> =
{"seq":22435808,"id":"1259540160F2016","changes":[{"rev":"2-7dcfd742f74c79=
286c3f3093595a83df"}],"deleted":true},
> =
{"seq":22435809,"id":"1259540640F2016","changes":[{"rev":"2-6bd122eb9f83c0=
838bc9875a1b73abaf"}],"deleted":true},
> =
{"seq":22435810,"id":"1259616780F2443","changes":[{"rev":"2-53e2311f5de705=
8fbfd55979816d3efc"}],"deleted":true},
> =
{"seq":22435811,"id":"1259616784F2443","changes":[{"rev":"2-caaff4cd1290f7=
807c2bcfeb6edc39e0"}],"deleted":true}
> ],
> "last_seq":22435811}
>=20
> This is a compacted copy of my main production database which is
> already on seq 106280009.
>=20
> When i try to purge I get a badarity error:
>=20
> curl -X POST 'localhost:5984/iris/_purge' -d
> '{"1259540160F2016":["2-7dcfd742f74c79286c3f3093595a83df"]}'
> {"error":"{{badarity,{#Fun<couch_db_updater.25.101160745>,\n
> [{2,<<124,230,79,165,16,199,208,127,32,211,160,223,180,12,3,28>>},\n
> {true,4185,19290621}]}},\n [{couch_key_tree,map_leafs_simple,3},\n
> {couch_key_tree,map_leafs_simple,3},\n  =
{couch_key_tree,map_leafs,2},\n
> {couch_db_updater,'-handle_call/3-fun-2-',2},\n  {lists,mapfoldl,3},\n
> {couch_db_updater,handle_call,3},\n  {gen_server,handle_msg,5},\n
> {proc_lib,init_p_do_apply,3}]}","reason":"{gen_server,call,\n
> [<0.28323.1>,\n             {purge_docs,[{<<\"1259540160F2016\">>,\n
> [{2,\n
> <<125,207,215,66,247,76,121,40,108,63,48,\n
> 147,89,90,131,223>>}]}]}]}"}
>=20
> This is what appears in my server log:
>=20
> [Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28323.1>] ** Generic =
server
> <0.28323.1> terminating=20
> ** Last message in was {purge_docs,[{<<"1259540160F2016">>,
>                                     [{2,
>=20
> <<125,207,215,66,247,76,121,40,108,
>                                         63,48,147,89,90,131,223>>}]}]}
> ** When Server state =3D=3D {db,<0.28322.1>,<0.28323.1>,nil,
>=20
> <<"1272901503155514">>,<0.28320.1>,<0.28324.1>,
>                            {db_header,5,106014165,0,
>                                {59608213341,{24516895,36227125}},
>                                {59608203141,60744020},
>                                {59608273696,[]},
>                                0,nil,nil,1000},
>                            106014165,
>                            {btree,<0.28320.1>,
>                                {59608213341,{24516895,36227125}},
>                                #Fun<couch_db_updater.7.132302543>,
>                                #Fun<couch_db_updater.8.107957134>,
>                                #Fun<couch_btree.5.124754102>,
>                                #Fun<couch_db_updater.9.46112288>},
>                            {btree,<0.28320.1>,
>                                {59608203141,60744020},
>                                #Fun<couch_db_updater.10.19027664>,
>                                #Fun<couch_db_updater.11.35033879>,
>                                #Fun<couch_btree.5.124754102>,
>                                #Fun<couch_db_updater.12.56344865>},
>                            {btree,<0.28320.1>,
>                                {59608273696,[]},
>                                #Fun<couch_btree.0.83553141>,
>                                #Fun<couch_btree.1.30790806>,
>                                #Fun<couch_btree.2.124754102>,nil},
>                            106014165,<<"iris">>,
>=20
> "/var/lib/couchdb/0.11.0/iris.couch",[],[],nil,
>                            {user_ctx,null,[],undefined},
>                            nil,1000,
>                            [before_header,after_header,on_file_open]}
> ** Reason for termination =3D=3D=20
> ** {{badarity,{#Fun<couch_db_updater.25.101160745>,
>               [{2,
>=20
> <<124,230,79,165,16,199,208,127,32,211,160,223,180,12,3,28>>},
>                {true,4185,19290621}]}},
>    [{couch_key_tree,map_leafs_simple,3},
>     {couch_key_tree,map_leafs_simple,3},
>     {couch_key_tree,map_leafs,2},
>     {couch_db_updater,'-handle_call/3-fun-2-',2},
>     {lists,mapfoldl,3},
>     {couch_db_updater,handle_call,3},
>     {gen_server,handle_msg,5},
>     {proc_lib,init_p_do_apply,3}]}
>=20
>=20
> [Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28323.1>]
> {error_report,<0.31.0>,
>    {<0.28323.1>,crash_report,
>     [[{initial_call,{couch_db_updater,init,['Argument__1']}},
>       {pid,<0.28323.1>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {{badarity,
>                    {#Fun<couch_db_updater.25.101160745>,
>                     [{2,
>=20
> <<124,230,79,165,16,199,208,127,32,211,160,223,180,12,
>                         3,28>>},
>                      {true,4185,19290621}]}},
>                [{couch_key_tree,map_leafs_simple,3},
>                 {couch_key_tree,map_leafs_simple,3},
>                 {couch_key_tree,map_leafs,2},
>                 {couch_db_updater,'-handle_call/3-fun-2-',2},
>                 {lists,mapfoldl,3},
>                 {couch_db_updater,handle_call,3},
>                 {gen_server,handle_msg,5},
>                 {proc_lib,init_p_do_apply,3}]},
>=20
> [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
>       {ancestors,
>=20
> [<0.28322.1>,couch_server,couch_primary_services,couch_server_sup,
>            <0.32.0>]},
>       {messages,[]},
>       {links,[<0.28322.1>]},
>       {dictionary,[]},
>       {trap_exit,false},
>       {status,running},
>       {heap_size,4181},
>       {stack_size,24},
>       {reductions,42618}],
>      []]}}
>=20
> [Mon, 03 May 2010 15:45:03 GMT] [error] [<0.28291.1>] Uncaught error =
in
> HTTP request: {exit,
>                                 {{{badarity,
>=20
> {#Fun<couch_db_updater.25.101160745>,
>                                     [{2,
>=20
> <<124,230,79,165,16,199,208,127,32,211,
>                                         160,223,180,12,3,28>>},
>                                      {true,4185,19290621}]}},
>                                   =
[{couch_key_tree,map_leafs_simple,3},
>                                    =
{couch_key_tree,map_leafs_simple,3},
>                                    {couch_key_tree,map_leafs,2},
>                                    {couch_db_updater,
>                                     '-handle_call/3-fun-2-',2},
>                                    {lists,mapfoldl,3},
>                                    {couch_db_updater,handle_call,3},
>                                    {gen_server,handle_msg,5},
>                                    {proc_lib,init_p_do_apply,3}]},
>                                  {gen_server,call,
>                                   [<0.28323.1>,
>                                    {purge_docs,
>                                     [{<<"1259540160F2016">>,
>                                       [{2,
>                                         <<125,207,215,66,247,76,121,
>                                           40,108,63,48,147,89,90,131,
>                                           223>>}]}]}]}}}
>=20
> [Mon, 03 May 2010 15:45:03 GMT] [info] [<0.28291.1>] Stacktrace:
> [{gen_server,call,2},
>             {couch_httpd_db,db_req,2},
>             {couch_httpd_db,do_db_req,2},
>             {couch_httpd,handle_request_int,5},
>             {mochiweb_http,headers,5},
>             {proc_lib,init_p_do_apply,3}]
>=20
> [Mon, 03 May 2010 15:45:03 GMT] [info] [<0.28291.1>] 127.0.0.1 - -
> 'POST' /iris/_purge 500
>=20
> Any suggestions would be greatly appreciated.
>=20
> Thanks,
>=20
> Mike
>=20
>=20
>=20
>=20