incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Compaction of a database with the same number of documents is getting slower over time
Date Tue, 22 Apr 2014 16:32:28 GMT
You can either run a filtered replication or block them with a validate_doc_update function
on the target database. Something like the following would allow you to delete docs on the
target but block the replication of tombstones:

function (newDoc, oldDoc, userCtx) {
    // any update to an existing doc is OK
    if(oldDoc) {
        return;
    }
    
    // reject tombstones for docs we don't know about
    if(newDoc[\"_deleted\"]) {
        throw({forbidden : \"We're rejecting tombstones for unknown docs\"})
    }
}

Regards, Adam

On Apr 22, 2014, at 10:22 AM, Boaz Citrin <bcitrin@gmail.com> wrote:

> You got it right.
> 
> How can I only replicate non deleted docs?
> בתאריך 22 באפר' 2014 17:15, "Adam Kocoloski" <kocolosk@apache.org> כתב:
> 
>> Sorry, just so we're super clear -- the "doc_count" is roughly constant
>> but the "doc_del_count" is rising? Compaction time scales with the sum of
>> those two values. Your options to get it back down under control at the
>> moment are
>> 
>> 1) Purge the deleted docs (lots of caveats about replication, potential
>> for view index resets, etc.)
>> 2) Rotate over to a new database, either by querying both or by
>> replicating non-deleted docs to the new one
>> 
>> Neither one is particularly palatable. CouchDB currently keeps the
>> tombstones around forever so that replication can always work. Making
>> changes on that front is a pretty subtle thing but maybe not completely
>> impossible.
>> 
>> Also, there's a new compactor in the works that is faster and generates
>> smaller files.
>> 
>> Adam
>> 
>> On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bcitrin@gmail.com> wrote:
>> 
>>> Yes, the documents change quickly, Many deletions and insertions, but
>> total
>>> number don't change that much.
>>> 
>>> 
>>> On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <
>> adam.kocoloski@gmail.com>wrote:
>>> 
>>>> To first order it shouldn't be the case.
>>>> 
>>>> Is the number of deleted documents growing with time? That'd have an
>>>> impact.
>>>> 
>>>> Adam
>>>> 
>>>>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bcitrin@gmail.com> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Our database contains more or less the same number of documents,
>> however
>>>>> the documents themselves change frequently.
>>>>> I would expect that compaction time will be the same, but I see that
>> over
>>>>> time it takes longer to compact the database.
>>>>> 
>>>>> Why is it so? Any way to overcome this?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Boaz
>>>> 
>> 
>> 


Mime
View raw message