Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of simon@cloudant.com designates
 74.125.82.172 as permitted sender)
Date: Mon, 11 Nov 2013 15:19:07 +0000
From: Simon Metson <simon@cloudant.com>
To: user@couchdb.apache.org
Message-ID: <56EA484C90F04E15B3D5F89581E8A6A6@cloudant.com>
In-Reply-To: 
 <CAMGHjAQVZ0+5Qzv_AzOt7nBDhByUed38TEpuZVrdYjkG0J+hAg@mail.gmail.com>
References: 
 <CAMGHjASjaVagGbvOGSJrPitoVVDN0qn7BxahT4y3t1XeRcHekg@mail.gmail.com>
 <527E548F.4020203@yandex.ru>
 <CABvT1DHMwHDAB4yc19h2z9avxfJOQ3Yp5FnQyBimULuGa5NJ_w@mail.gmail.com>
 <CAMGHjATWVSc4WyJ4N3g-roGkPKMxEF_DwmXieuy7qS3wugTYpA@mail.gmail.com>
 <CAK0S_BDsCRwUXBSUtjr-rBMXJuw_JcdhdTDLKUYckPj5+BC7XA@mail.gmail.com>
 <CABvT1DEnX0He=ZvYF5ucHcAJGphwso2vajS8jSd6fBLtzLA0Ag@mail.gmail.com>
 <1CD5C733-4854-407A-A8DF-8E29A1A0493B@couchbase.com>
 <CAMGHjARqtzUWujNn6EW-v1BAk=XYSp3_Ru8VNwgmeYb_M=m8Zw@mail.gmail.com>
 <CAKiq=tATz_ymdvX7mfGowM2xY82jniz1_-RjJ5Dj7gGKZP43AA@mail.gmail.com>
 <CAMGHjAT2GQYjmRp=ntDRAMZGGAFtRbKfUVW0zU+mCY6yh=BSjQ@mail.gmail.com>
 <CAMGHjAQ4qqu=eHm31L7T9fDN90Fiit3rjMGQOprF+7DsvN8rxw@mail.gmail.com>
 <CAL+Y1nt=RtTtCQ07_0OYHhbk9ESUhhDuShedHbgYcvZvHDj_Zw@mail.gmail.com>
 <CAMGHjAQWKtbrjDEWyrW3n1DiPAdCLsS8tWOdXV6-iJ5XbOAY1g@mail.gmail.com>
 <D0BF25869D534E1EBA00B83F3EB7E34F@cloudant.com>
 <CAMGHjARDXkE2RwLQ9O=weg4CQRzY4bzpUin-u0gwxVJTnk0z2Q@mail.gmail.com>
 <CAMGHjAQVZ0+5Qzv_AzOt7nBDhByUed38TEpuZVrdYjkG0J+hAg@mail.gmail.com>
Subject: Re: Storage limitations?
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

I think if you can happily fit it all into CouchDB (and from the numbers =
you=E2=80=99ve mentioned so far I=E2=80=99m confident you can) I=E2=80=99=
d go with that - simplest solution. If your 1000=E2=80=99s become 1000000=
=E2=80=99s then you can bite the bullet and switch to metadata in CouchDB=
, image data in something else. =20


On Monday, 11 November 2013 at 15:16, Mark Deibert wrote:

> =40Simon: I understand what your saying about =22source of truth=22. Wh=
at is
> master record, the files or the Couch records if the upload succeeds an=
d
> the Couch record write fails or vise-versa. Not sure what the answer is=

> here either.
> =20
> =20
> On Mon, Nov 11, 2013 at 10:00 AM, Mark Deibert <mark.deibert=40gmail.co=
m (mailto:mark.deibert=40gmail.com)>wrote:
> =20
> > =40Simon: Thanks for the advice. As I type, I don't anticipate any
> > aggregation on the Photo docs, but I should think about that some mor=
e.
> > =20
> > =20
> > On Mon, Nov 11, 2013 at 9:56 AM, Simon Metson <simon=40cloudant.com (=
mailto:simon=40cloudant.com)> wrote:
> > =20
> > > Hey,
> > > I=E2=80=99d go with what you=E2=80=99re doing (1:1 doc:photo).
> > > =20
> > > 2 seems a bit weird - do you mean host the app in a different serve=
r to
> > > the data or have different databases for different types of data (t=
hat
> > > might work, if you never want to query across types of data)=3F 3 m=
ight work
> > > so long as you never want to aggregate across the categories. 4 is =
a
> > > reasonable approach for very very large attachments, but you can ge=
t into
> > > consistency issues - what=E2=80=99s the source of truth the databas=
e or the
> > > filesystem=3F
> > > Cheers
> > > Simon
> > > =20
> > > =20
> > > On Monday, 11 November 2013 at 14:41, Mark Deibert wrote:
> > > =20
> > > > A followup on the =221000's of images=22 question. I could approa=
ch this a
> > > > couple ways. Currently each image is attached to it's own Photo d=
oc.
> > > =20
> > > =20
> > > Which
> > > > I've read is better for replication than one attachment with many=

> > > > attachments. So that's fine, but will Couch have any issue managi=
ng
> > > =20
> > > =20
> > > several
> > > > thousand of these Photo docs, each with a 3MB'ish image attachmen=
t=3F If
> > > =20
> > > =20
> > > you
> > > > were building this Couchapp, would you...
> > > > =20
> > > > 1) Keep the photos as described above in one CouchDB
> > > > 2) Move the Photo docs with attachments out into a separate Couch=
DB
> > > > 3) Do 2, but break Photos into multiple categorized CouchDBs
> > > > 4) Upload the images to the filesystem, just store the link in Co=
uch
> > > > =20
> > > > I want to build this Couchapp in such a way as to not make life
> > > miserable
> > > > for CouchDB :-D
> > > > =20
> > > > =20
> > > > =20
> > > > =20
> > > > On Sun, Nov 10, 2013 at 6:33 PM, Dave Cottlehuber <dch=40jsonifie=
d.com (mailto:dch=40jsonified.com)(mailto:
> > > dch=40jsonified.com (mailto:dch=40jsonified.com))> wrote:
> > > > =20
> > > > > On 10 November 2013 23:14, Mark Deibert <mark.deibert=40gmail.c=
om (mailto:mark.deibert=40gmail.com)(mailto:
> > > mark.deibert=40gmail.com (mailto:mark.deibert=40gmail.com))> wrote:=

> > > > > > I read an article somewhere that using include=5Fdocs is =22h=
ard=22 on
> > > > > =20
> > > > =20
> > > =20
> > > =20
> > > memory
> > > > > =20
> > > > > =20
> > > > > or
> > > > > > disk or in some way taxes Couch and therefore you should just=
 emit
> > > > > =20
> > > > =20
> > > =20
> > > =20
> > > the
> > > > > =20
> > > > > =20
> > > > > doc.
> > > > > > Is this true=3F
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > Like most general statements it has some truth and some lies in=
 it :-)
> > > > > =20
> > > > > views and docs are stored in separate .couch btree files on dis=
k.
> > > > > =20
> > > > > emit(key, doc) puts a full copy of the doc (that's already in t=
he doc
> > > > > .couch b=7Etree) into the view b=7Etree.
> > > > > =20
> > > > > advantage - no need to hop over to the doc .couch file to retri=
eve the
> > > > > document.
> > > > > disadvantage - you now have 2 copies of the doc in separate fil=
es,
> > > > =20
> > > =20
> > > =20
> > > wasted
> > > > > space.
> > > > > =20
> > > > > If you do things right, and your app fits this model, the gener=
ated
> > > > > etags from views and docs can be cached in nginx or similar, an=
d
> > > > > repeated queries don't need to hit your couch.
> > > > > =20
> > > > > So yes, include=5Fdocs means extra reads, but like most of thes=
e things
> > > > > you should benchmark your situation, under a realistic load, no=
t just
> > > > > pumping 1000 single-doc reads at it.
> > > > > =20
> > > > > A+
> > > > > Dave
> > > > =20
> > > =20
> > =20
> =20