couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz <bcit...@gmail.com>
Subject Re: Compaction of a database with the same number of documents is getting slower over time
Date Tue, 22 Apr 2014 16:51:21 GMT
Thanks Adam!

> On 22 באפר 2014, at 19:32, Adam Kocoloski <kocolosk@apache.org> wrote:
> 
> You can either run a filtered replication or block them with a validate_doc_update function
on the target database. Something like the following would allow you to delete docs on the
target but block the replication of tombstones:
> 
> function (newDoc, oldDoc, userCtx) {
>    // any update to an existing doc is OK
>    if(oldDoc) {
>        return;
>    }
> 
>    // reject tombstones for docs we don't know about
>    if(newDoc[\"_deleted\"]) {
>        throw({forbidden : \"We're rejecting tombstones for unknown docs\"})
>    }
> }
> 
> Regards, Adam
> 
>> On Apr 22, 2014, at 10:22 AM, Boaz Citrin <bcitrin@gmail.com> wrote:
>> 
>> You got it right.
>> 
>> How can I only replicate non deleted docs?
>> בתאריך 22 באפר' 2014 17:15, "Adam Kocoloski" <kocolosk@apache.org>
כתב:
>> 
>>> Sorry, just so we're super clear -- the "doc_count" is roughly constant
>>> but the "doc_del_count" is rising? Compaction time scales with the sum of
>>> those two values. Your options to get it back down under control at the
>>> moment are
>>> 
>>> 1) Purge the deleted docs (lots of caveats about replication, potential
>>> for view index resets, etc.)
>>> 2) Rotate over to a new database, either by querying both or by
>>> replicating non-deleted docs to the new one
>>> 
>>> Neither one is particularly palatable. CouchDB currently keeps the
>>> tombstones around forever so that replication can always work. Making
>>> changes on that front is a pretty subtle thing but maybe not completely
>>> impossible.
>>> 
>>> Also, there's a new compactor in the works that is faster and generates
>>> smaller files.
>>> 
>>> Adam
>>> 
>>>> On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bcitrin@gmail.com> wrote:
>>>> 
>>>> Yes, the documents change quickly, Many deletions and insertions, but
>>> total
>>>> number don't change that much.
>>>> 
>>>> 
>>>> On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <
>>> adam.kocoloski@gmail.com>wrote:
>>>> 
>>>>> To first order it shouldn't be the case.
>>>>> 
>>>>> Is the number of deleted documents growing with time? That'd have an
>>>>> impact.
>>>>> 
>>>>> Adam
>>>>> 
>>>>>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bcitrin@gmail.com>
wrote:
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> Our database contains more or less the same number of documents,
>>> however
>>>>>> the documents themselves change frequently.
>>>>>> I would expect that compaction time will be the same, but I see that
>>> over
>>>>>> time it takes longer to compact the database.
>>>>>> 
>>>>>> Why is it so? Any way to overcome this?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Boaz
>>>>> 
>>> 
>>> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message