Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
From: Adam Kocoloski <kocolosk@apache.org>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: Store meta data of millions of images for search purpose
Date: Wed, 14 Jun 2017 13:17:54 -0400
References: <48B90C5F0B65C944A52392D05C410EE031F6D070@genmail2010.vitalhuber.local>
 <CADsFwidyfNBC+WRmp9LLgF-Z4yQ7DFEzWFoKOvS8yVVbXy+fWw@mail.gmail.com>
 <48B90C5F0B65C944A52392D05C410EE031F6D088@genmail2010.vitalhuber.local>
To: user@couchdb.apache.org
In-Reply-To: <48B90C5F0B65C944A52392D05C410EE031F6D088@genmail2010.vitalhuber.local>
Message-Id: <8C00C38E-061E-4886-BEDA-53D3BF315F1A@apache.org>
archived-at: Wed, 14 Jun 2017 17:18:01 -0000

Hi Ajay, the view engine will happily keep up with 10k - 20k updates per =
day. If you=E2=80=99re using CouchDB 2.0 you can distribute this =
database across several underlying physical shards. You won=E2=80=99t =
need to do that just to keep up with your designed update rate, but an =
index with a billion entries will be easier to manage operationally if =
it=E2=80=99s sharded. Compaction in particular can be an unwieldy =
operation on an index that large. Cheers,

Adam

> On Jun 14, 2017, at 8:47 AM, Ajay Pawaskar <APawaskar@genesisinfo.com> =
wrote:
>=20
> as per application there will be multiple images per =
record[201700000000002...]. images can be of different types [current, =
viewable] like one image is marked as bCurrent=3Dtrue and another with =
bCurrent=3Dfalse. then there will be search where I need to search image =
related to record which have bCurrent=3Dtrue/false. if I make documents =
per image then number of documents will be increased [more than =
billions]
>=20
> -----Original Message-----
> From: aa mm [mailto:assaf.morami@gmail.com]=20
> Sent: Wednesday, June 14, 2017 6:11 PM
> To: user@couchdb.apache.org
> Subject: Re: Store meta data of millions of images for search purpose
>=20
> What each document represents? Why do you need to generate ids? Is =
this a requirement?
>=20
> If not, and image file name is unique, then you can make it so each =
document represents an image. _id will be the image file name, and thus =
you won't need a view to access an image, you'll need only the image =
name.
>=20
> Assaf.
>=20
>=20
> =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 14 =D7=91=D7=99=D7=95=D7=A0=D7=99 =
2017 01:13 PM,=E2=80=8F "Ajay Pawaskar" <APawaskar@genesisinfo.com>
> =D7=9B=D7=AA=D7=91:
>=20
> Hi,
> I am having question related to storing millions/billions of document =
in Couch DB and use view to get required documents. I would like to know =
about performance/scalability of views/ Couch DB in following case.
>=20
> I have  application where I need to store meta data of =
millions/billions of images for search purpose. Images will be =
added/updated/deleted/retrieve on regular basis [10000/20000 per day].
>=20
> we are thinking to store these documents in following format e.g.
> {
>   "_id": "201700000000002", /* this will be generated by our =
application*/
>   "_rev": "1-b85e805bdd293a5f727517beea9512b3",
>   "12398712397129": {"bCurrent": true, "bCanView": true} =
/*"12398712397129" is image file name*/
>   "98127397192319": {"bCurrent": false, "bCanView": false}} =
/*"98127397192319" is image file name*/
>=20
> }
>=20
> {
>   "_id": "201700000000003", /* this will be generated by our =
application*/
>   "_rev": "1-b85e83432d293a5f727517beea9512b3",
>   "89723979823929": {"bCurrent": true, "bCanView": true} =
/*"12398712397129" is image file name*/
>   "92347324667324": {"bCurrent": false, "bCanView": false}} =
/*"98127397192319" is image file name*/
>   "72832532467217": {"bCurrent": true, "bCanView": false}} =
/*"72832532467217" is image file name*/ }
>=20
>=20
> so if user want to get current image for record  201700000000002 we =
will be having following view
>=20
> function(doc) {
> for(var prop in doc){
> if(prop!=3D"_id" && prop!=3D"_rev"){
>  if(doc[prop].bCurrent!=3D=3Dundefined && doc[prop].bCurrent){
>     emit(doc._id, { RecordID: doc._id,ImageID: prop,bCurrent:
> doc[prop].bCurrent, doc[prop].bCanView}); } } } which will be called =
with key "201700000000002".
>=20
> but as mentioned earlier Images will be added/updated/deleted/retrieve =
on regular basis [10000/20000 per day] how this going to affect views =
performance?
>=20
> Regards,
> Ajay.