Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of stemail23@gmail.com
 designates 209.85.192.170 as permitted sender)
References: <4452E2B7937E5944BDA554F6A6D5EB363DDA6218@abn-exch1b.green.sophos>
 <CACLy94WeX6mp+AS3HM0J8tONsFy8uVuHNZNAqwr=kKmNdxnCHQ@mail.gmail.com>
 <CABvT1DE3FTG9ZKCjeSvF8YuiP5jtUE1g=p+vhrOgYY9gyffQEw@mail.gmail.com>
 <CACLy94Xgp6GtVRuSsO+qJf30AXZQzmXUx89Aa0JXYHp7dQtrxg@mail.gmail.com>
From: Steven Barlow <stemail23@gmail.com>
Mime-Version: 1.0 (1.0)
In-Reply-To: 
 <CACLy94Xgp6GtVRuSsO+qJf30AXZQzmXUx89Aa0JXYHp7dQtrxg@mail.gmail.com>
Date: Fri, 5 Jul 2013 21:52:19 +1000
Message-ID: <2230614989236312791@unknownmsgid>
Subject: Re: Purging documents and view invalidation
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Sorry if this is a tangent, but I wanted to pick up on the "rarely
used in the wild" thread: I personally intend to use purge, because I
have temporary partial (filtered) replications of a "master" database
at remote sites. When the data has been consumed by the remote site, I
figured I could purge it (to save space). Is this not a valid, or
common use case for purging?

On 05/07/2013, at 7:21 PM, Jason Smith <jason.h.smith@gmail.com> wrote:

> I slightly disagree with Bob, but he is right that all purge buys you (vs.
> filtered replication and then swapping DBs) is a little bit of uptime.
> Purge is not "untested" but it is rarely used in the wild, so the
> cost/benefit for your uptime is something between "risky" and "unknown."
>
> (For me, personally, I would purge.)
>
>
> On Fri, Jul 5, 2013 at 3:31 PM, Robert Newson <rnewson@apache.org> wrote:
>
>> Paul,
>>
>> If you replicate this database to another database and use a filter
>> that blocks deleted documents, the target will not contain a trace of
>> your 100 million deletes (that is, you can build a new database
>> without cruft without messing with your existing database). During the
>> replication, you can query the view on the target to build it
>> incrementally, or wait till the end, query it once and wait for
>> completion. At the end, flip your app to look at the new database
>> instead.
>>
>> The _purge feature is really only for the case where you accidentally
>> write your root password down in a document id or something (since
>> compaction will sweep away old document contents). I advise against
>> using it for any other reason.
>>
>> B.
>>
>>
>>
>>
>> On 5 July 2013 09:17, Jason Smith <jhs@apache.org> wrote:
>>> Hi, Paul. I wrote up some thoughts on purging here:
>>> https://github.com/iriscouch/cqs#purging-couchdb
>>>
>>> Note, that procedure is untested. It works as a thought experiment only.
>>>
>>> The procedure looks complicated, but all you will need is the core purge,
>>> view, purge, view, etc. cadence as described in Damien's email I linked
>> to.
>>> As long as you never purge twice before hitting the view, you are fine.
>>> Again, to my knowledge, the purge code is less well tested than other
>> parts
>>> of CouchDB, so perhaps copy your .couch file and try with that until you
>>> are confident.
>>>
>>>
>>> On Fri, Jul 5, 2013 at 2:37 PM, Paul Hirst <paul.hirst@sophos.com>
>> wrote:
>>>
>>>> I would like to purge a few (~100 million) documents from my database.
>>>> I've been going through deleting them all, and that'll be complete in
>> the
>>>> next few days but I would like to free up some extra space by purging
>> them
>>>> also.
>>>>
>>>> My concern is around a comment on the wiki page here
>>>> http://wiki.apache.org/couchdb/Purge_Documents
>>>>
>>>> 'If you have purged more than one document between querying your views,
>>>> you will find that they will rebuild from scratch.'
>>>>
>>>> Since I have already deleted the documents I know they aren't showing up
>>>> in the view any longer. Is there any way I can avoid this view
>>>> invalidation? (My views take about 10 days to build from scratch so I
>> can't
>>>> afford the hit).
>>>>
>>>> I have a replica of the database. I could do the purge on the replica,
>>>> wait for the view to rebuild, switch over, purge on the original db,
>> wait
>>>> for the view, switch back, unless there are any obvious problems with
>> this
>>>> approach?
>>>>
>>>> Cheers,
>>>> Paul
>>>>
>>>> ________________________________
>>>>
>>>> Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP,
>>>> United Kingdom.
>>>> Company Reg No 2096520. VAT Reg No GB 991 2418 08.
>>