Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of fdmanana@gmail.com designates
 74.125.83.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=DGTLUsWX1dSuthwcdwAyCEwVv4TVhnhK2e6sv+Szs9DLHq/TL6Z1nCSBJpWtE/CzJ3
         hGR1TCsyatBi1TnYq3FUBQiTGE/wDJQq2Je1nndxQGwxZGpRGxqw+mRNgXFrkFHfnAnE
         +CM104RkUQ2KkwVTX/4lnWWDdEGbGUgbdJwFM=
MIME-Version: 1.0
Sender: fdmanana@gmail.com
In-Reply-To: <alpine.DEB.2.00.1104080931330.22381@steve-laptop.beatport.com>
References: <alpine.DEB.2.00.1104071520510.19227@steve-laptop.beatport.com>
	<BANLkTikVEsvuaxwM67JMdNk1BYw8iO4WEg@mail.gmail.com>
	<alpine.DEB.2.00.1104080931330.22381@steve-laptop.beatport.com>
Date: Fri, 8 Apr 2011 16:38:55 +0100
Message-ID: <BANLkTikpC4dKBRKsfh489Hj0L2Lop05jTQ@mail.gmail.com>
Subject: Re: Couch Replication: slave on-disk size bloat
From: Filipe David Manana <fdmanana@apache.org>
To: steve.webb@beatport.com
Cc: user@couchdb.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 8, 2011 at 4:31 PM,  <steve.webb@beatport.com> wrote:
> I'm not sure what's in the databases. =C2=A0Why? =C2=A0Would large attach=
ments explain
> this in some way?

Yes.
The current replicator in 1.0.x and 1.1 (yet to be released), when
replicating a new revision of a document with attachments, it
replicates all the attachments, not just the attachments missing on
the target. So you might end up with the same attachments several
times in the target database file (compaction will fix this of
course).

The new replicator, in trunk only, doesn't have this behaviour - it
replicates only the missing attachments.

>
> - Steve
>
> On Fri, 8 Apr 2011, Filipe David Manana wrote:
>
>> Do you have many and/or large attachments?
>>
>> On Thu, Apr 7, 2011 at 10:24 PM, =C2=A0<steve.webb@beatport.com> wrote:
>>>
>>> Hello there.
>>>
>>> New couch admin here.
>>>
>>> I've got two couchdb machines that are not being written to.
>>>
>>> I cleaned off the slave, then replicated and compacted the DBs on the
>>> slave
>>> and the slave DB size on-disk is *way* more than on the master.
>>>
>>> Also, we've found that when the slaves are being actively written
>>> (replicated) to, over time, they grow > 10x the size of the master's fi=
le
>>> size.
>>>
>>> Master:
>>>
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0 =C2=A0 =C2=A020565 Apr =C2=A0=
7 12:02 api-keys.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb 2414678118 Apr =C2=A07 11:41
>>> similar-tracks.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0851714150 Apr =C2=A07 11:52 si=
temap.couch
>>> drwxr-xr-x =C2=A02 couchdb couchdb =C2=A0 =C2=A0 =C2=A0 4096 Feb 15 01:=
10 .sitemap_design
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0 76939362 Apr =C2=A07 11:27 to=
p-downloads.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0268513382 Apr =C2=A07 11:30
>>> users-also-bought.couch
>>>
>>> Slave (after delete, replication and compaction):
>>>
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0 =C2=A0 =C2=A0 8280 Apr =C2=A0=
7 12:02 api-keys.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb 5261164653 Apr =C2=A07 14:16
>>> similar-tracks.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0860319850 Apr =C2=A07 13:58 si=
temap.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0122777700 Apr =C2=A07 13:50 to=
p-downloads.couch
>>> -rw-r--r-- =C2=A01 couchdb couchdb =C2=A0285810794 Apr =C2=A07 13:52
>>> users-also-bought.couch
>>>
>>> Q1: Is this normal?
>>> Q2: Is there a way to fix this or turn off whatever is turned on that
>>> causes
>>> this?
>>>
>>> - Steve Webb
>>>
>>> --
>>> Steve Webb | System Administrator
>>> Beatport | Play With Music
>>> ------------------------------------------
>>> 2399 Blake Street, Suite 170
>>> Denver, Colorado USA 80205
>>> tel: +1.720.932.9103
>>> fax: +1.720.932.9104
>>> noc: +1.303.565.2710
>>> mobile: +1.303.564.4269
>>>
>>
>>
>>
>>
>
> --
> Steve Webb | System Administrator
> Beatport | Play With Music
> ------------------------------------------
> 2399 Blake Street, Suite 170
> Denver, Colorado USA 80205
> tel: +1.720.932.9103
> fax: +1.720.932.9104
> noc: +1.303.565.2710
> mobile: +1.303.564.4269


--=20
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
=C2=A0Unreasonable men adapt the world to themselves.
=C2=A0That's why all progress depends on unreasonable men."