Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of bcitrin@gmail.com designates
 209.85.192.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACLy94VHX6GqQv_2v1ZgY674WoV8PzLLQSWz1_vV+7XvGUyVdQ@mail.gmail.com>
References: 
 <CAOxguM0LspwwGbBLntVu5rZgnK9qsuk8o6XcyqnwnBy3Y+S-yA@mail.gmail.com>
	<4AD73424-F7BB-49F4-ABC9-43A88D34BDCA@couchbase.com>
	<CACLy94UmJDqDg3JjHisUxJLPtJoZxp9MyxdJvaNW+p67rCp5wQ@mail.gmail.com>
	<CAK0S_BDS1-Z7mHRHF6vD0De77iB5z-u4oQt+SQgkormSFZMjbA@mail.gmail.com>
	<DB2C717B-928D-4EF9-B9E2-D872EA725041@couchbase.com>
	<CACLy94VHX6GqQv_2v1ZgY674WoV8PzLLQSWz1_vV+7XvGUyVdQ@mail.gmail.com>
Date: Fri, 31 Jan 2014 21:27:25 +0200
Message-ID: 
 <CAOxguM1wchf13xMMOfTr8hWJQB-=WmU_NvF4Nqhjspt_UkaDpA@mail.gmail.com>
Subject: Re: Replication vs. Compaction
From: Boaz Citrin <bcitrin@gmail.com>
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

@Jason - the purpose is that compaction will not catch up with heavy
write rate and will affect performance.
So I replicate/copy to a side database, compact it, then replicate the
new changes from the original and switchover.
I wonder if I can omit the compaction if replication gives the same result.
Will replication remove old revisions too?


On Fri, Jan 31, 2014 at 9:08 PM, Jason Smith <jhs@apache.org> wrote:
> Anyway I think the broader point is, compaction is for compacting databases
> (removing old document revisions), and replication is for making a copy of
> a database (or subset). If compaction is causing downtime then that is a
> different bug to talk about, but it should be totally transparent.
>
> Jens (incidentally it's nice to talk with you again): the compactor will
> notice that it has not caught up yet, and it will run again from the old
> "end" to the real end. Of course, there may be changes during that run too,
> so it will repeat. Usually each iteration has a much, much smaller window.
> In practice, you tend to see one "not caught up" message in the logs, and
> then it's done. However there is a pathological situation where you are
> updating faster than the compactor can run, and you will get an infinite
> loop (plus very heavy i/o and filesystem waste as the compactor is
> basically duplicating your .couch into a .couch.compact forever).
>
>
> On Sat, Feb 1, 2014 at 12:59 AM, Jens Alfke <jens@couchbase.com> wrote:
>
>>
>> On Jan 31, 2014, at 9:46 AM, Mark Hahn <mark@hahnca.com> wrote:
>>
>> > It wouldn't matter if it did.  Within the same server linux
>> short-circuits
>> > http to make it the same as unix sockets, i.e. very little overhead.
>>
>> I think you mean it short-circuits TCP :)
>> There's extra work involved in HTTP generation & parsing no matter what
>> transport you're sending it over. And then the replicator is doing a bunch
>> of JSON and multipart generation/parsing.
>> Whereas the compactor, I would imagine, is mostly just making raw
>> read/write calls while walking the b-tree.
>>
>> Anyway; this makes me wonder what happens when changes are made to a
>> database during compaction. The compaction processes working off of a
>> snapshot of the database from the point that it started, so it's not going
>> to copy over new changes. Does that mean they get lost, or does the
>> compactor have extra smarts to run a second phase where it copies over all
>> revs created since the snapshot?
>>
>> --Jens