incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <rnew...@apache.org>
Subject Re: Purging documents and view invalidation
Date Fri, 05 Jul 2013 08:31:21 GMT
Paul,

If you replicate this database to another database and use a filter
that blocks deleted documents, the target will not contain a trace of
your 100 million deletes (that is, you can build a new database
without cruft without messing with your existing database). During the
replication, you can query the view on the target to build it
incrementally, or wait till the end, query it once and wait for
completion. At the end, flip your app to look at the new database
instead.

The _purge feature is really only for the case where you accidentally
write your root password down in a document id or something (since
compaction will sweep away old document contents). I advise against
using it for any other reason.

B.




On 5 July 2013 09:17, Jason Smith <jhs@apache.org> wrote:
> Hi, Paul. I wrote up some thoughts on purging here:
> https://github.com/iriscouch/cqs#purging-couchdb
>
> Note, that procedure is untested. It works as a thought experiment only.
>
> The procedure looks complicated, but all you will need is the core purge,
> view, purge, view, etc. cadence as described in Damien's email I linked to.
> As long as you never purge twice before hitting the view, you are fine.
> Again, to my knowledge, the purge code is less well tested than other parts
> of CouchDB, so perhaps copy your .couch file and try with that until you
> are confident.
>
>
> On Fri, Jul 5, 2013 at 2:37 PM, Paul Hirst <paul.hirst@sophos.com> wrote:
>
>> I would like to purge a few (~100 million) documents from my database.
>> I've been going through deleting them all, and that'll be complete in the
>> next few days but I would like to free up some extra space by purging them
>> also.
>>
>> My concern is around a comment on the wiki page here
>> http://wiki.apache.org/couchdb/Purge_Documents
>>
>> 'If you have purged more than one document between querying your views,
>> you will find that they will rebuild from scratch.'
>>
>> Since I have already deleted the documents I know they aren't showing up
>> in the view any longer. Is there any way I can avoid this view
>> invalidation? (My views take about 10 days to build from scratch so I can't
>> afford the hit).
>>
>> I have a replica of the database. I could do the purge on the replica,
>> wait for the view to rebuild, switch over, purge on the original db, wait
>> for the view, switch back, unless there are any obvious problems with this
>> approach?
>>
>> Cheers,
>> Paul
>>
>> ________________________________
>>
>> Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP,
>> United Kingdom.
>> Company Reg No 2096520. VAT Reg No GB 991 2418 08.
>>

Mime
View raw message