Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.210.192 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ISe6dmAHmmQ7QmCOPMZ1koX7uP+p7EJMzy4hf5enMSo8BwmHyIJ14vCkJ9PN2ggmPK
         Ggbgre5rGMxuvE00YU9gYnOxdcXsayEEw3Kvvo73MMXhm4KXHZ6VbIg75gZRgC2AfVWj
         bhxy1+qvQz8VNE8crcZN8l6ucWS5VZhn+NT1Q=
MIME-Version: 1.0
In-Reply-To: <6cfccb3d0910141648p3dd7f56cs55a0645d47cee0a@mail.gmail.com>
References: <F97DC3C5-3430-4014-95F1-4570389A2077@metaweb.com>
	 <e2111bbb0910122151h220a25f5v75f9ccb13f4b42b4@mail.gmail.com>
	 <9FAB7E63-1917-4ABF-88BE-214328DC7EA9@metaweb.com>
	 <e2111bbb0910130537h7c1a64fau8b85bd9866349111@mail.gmail.com>
	 <83735A1C-46EE-419E-8D3F-E2F7839F7D22@metaweb.com>
	 <e282921e0910131559m29a62eb8n771e62e642090396@mail.gmail.com>
	 <6cfccb3d0910141648p3dd7f56cs55a0645d47cee0a@mail.gmail.com>
Date: Wed, 14 Oct 2009 19:53:50 -0400
Message-ID: <e2111bbb0910141653k631449c0wd5a3f3c4114079a8@mail.gmail.com>
Subject: Re: insert performance
From: Paul Davis <paul.joseph.davis@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Glenn,

CouchDB will already create subdirectories for db's with /'s. Though
its not as automatic it'll allow for clients to do exactly the
structure you're thinking of.

HTH,
Paul

On Wed, Oct 14, 2009 at 7:48 PM, Glenn Rempe <glenn@rempe.us> wrote:
> Millions op DB's?
>
> Wouldn't you run into filesystem limitations due to the fact that
> CouchDB writes all of its DB's/indexes into a single dir?
>
> e.g For the ext3 filesystem "There is a limit of 31998 sub-directories
> per one directory, stemming from its limit of 32000 links per inode."
>
> http://en.wikipedia.org/wiki/Ext3
>
> And my limited knowledge of filesystem internals says that the more
> files you have in a single dir the longer it will take to seek on
> those files.
>
> "The ext2 inode specification allows for over 100 trillion files to
> reside in a single directory, however because of the current
> linked-list directoryimplementation, only about 10-15 thousand files
> can realistically be stored in a single directory. =A0This is why
> systems such as Squid (http://www.squid-cache.org ) use cache
> directories with many subdirectories - searching through tens of
> thousands of files in one directory is sloooooooow."
>
> http://answers.google.com/answers/threadview/id/122241.html
>
> Of course this will vary by filesystem in absolute terms, but I think
> the concept is the same for all current file systems. No?
>
> CouchDB might really be able to address this if it did something like
> make subdirs under the couchdb data dir that were derived from
> portions of a hash of the filename. =A0Using such a 2 or 3 level deep
> dir structure would indeed allow for a huge number of DB's.
>
> e.g. if the db name hashes to a123df4g34fd.couch
>
> Make dirs/files like:
>
> DATA_DIR/a1/12/3d/f4g34fd.couch =A0# DB
> DATA_DIR/a1/12/3d/.some_view_index_hidden_dir =A0# view index
>
> No?
>
> On Tue, Oct 13, 2009 at 3:59 PM, Chris Anderson <jchris@apache.org> wrote=
:
>> On Tue, Oct 13, 2009 at 6:29 AM, Brian Karlak <zenkat@metaweb.com> wrote=
:
>>>
>>> One caveat, however: we have one (somewhat funky) usecase which creates=
 a
>>> large number of small databases. =A0Could the existence of several thou=
sand
>>> small databases affect performance?
>>>
>>
>> We definitely support the many-databases use case (eg: 1 per user, aka
>> millions of databases). I think there is extra support for that in
>> 0.9.1, and of course 0.10 has only improved from there.
>>
>> Chris
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>
>
>
> --
> Glenn Rempe
>
> email =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : glenn@rempe.us
> voice =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : (415) 894-5366 or (415)-89G-LENN
> twitter =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: @grempe
> contact info =A0 =A0 =A0 =A0: http://www.rempe.us/contact.html
> pgp =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: http://www.rempe.us/gnupg.tx=
t
>