Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of jens@couchbase.com designates
 206.225.164.30 as permitted sender)
From: Jens Alfke <jens@couchbase.com>
To: "user@couchdb.apache.org" <user@couchdb.apache.org>,
	"matthias.eck@gmail.com" <matthias.eck@gmail.com>
Date: Fri, 12 Jul 2013 12:03:15 -0700
Subject: Re: Database too large for partition
Thread-Topic: Database too large for partition
Thread-Index: Ac5/MnZZCPI4k4qMRp+w2mrJ0btDkQ==
Message-ID: <2E279AEA-D6E5-4DBB-A599-288E2FBCB2A8@couchbase.com>
References: 
 <CADD5XYJ21QTw3BGWEgF1zQ6ox52NNY6A8Ns_AE0J9U3KsSUmDw@mail.gmail.com>
 <CAGxYWpbWdC4RmVkP3kpbd0D_XXMnhcbk=Rhh6oV_2WqhFudf9g@mail.gmail.com>
 <CABvT1DHivYvmKjY7JbrTc7GffAMM3uxbb3uaYqnLnEVH6WkFQA@mail.gmail.com>
 <CABvT1DF-hJh4PS35tj7TdOfgyt4qcvp6gSkKUfhYwhMC9HbrhA@mail.gmail.com>
 <CADD5XY+-Jx286PGX=4FiTXT_ky2x_4O9OyJD_VkN1S+MDgBdwg@mail.gmail.com>
In-Reply-To: 
 <CADD5XY+-Jx286PGX=4FiTXT_ky2x_4O9OyJD_VkN1S+MDgBdwg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Jul 11, 2013, at 12:20 PM, Matthias Eck <matthias.eck@gmail.com> wrote:

> The database file itself has about 500GB, the views have another 500GB.

Is the database 500GB of pure JSON, or are there attachments? (The idea of =
500GB of JSON boggles my mind, but then, I=92m not a big-data guy.)

I ask because if a lot of that size is attachments, you can save space by s=
toring those elsewhere, like in S3, and leaving just URLs or other IDs in t=
he docs. This will also speed up compaction because the attachment data doe=
sn=92t have to be copied. The drawback is that you=92ll have to manually de=
lete attachments after their corresponding doc[s] are deleted or updated.

=97Jens=