Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: <BANLkTi=nzdO1TKVdkRgY95WhtTpp6fyw1g@mail.gmail.com>
References: <BANLkTi=3emYZ-Vimrqe2m3GQdysOJ_LOYQ@mail.gmail.com>
	<BANLkTikyrvO4vNH=FJXQr0gut1+4k0O_hQ@mail.gmail.com>
	<BANLkTi=nzdO1TKVdkRgY95WhtTpp6fyw1g@mail.gmail.com>
Date: Fri, 1 Jul 2011 09:09:18 +0100
Message-ID: <BANLkTim0vRRkr4_SossXAv4MLe=UNGsh3w@mail.gmail.com>
Subject: Re: Frugal Erlang vs Resources Hungry CouchDB
From: Robert Newson <rnewson@apache.org>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Every header is potentially used but every header supplants every
previous header. The reason we must write a header for each update is
durability. If we did not do so, then a crash could undo an update,
which is not acceptable for a database. Moreover, with the recommended
production setting of delayed_commits=3Dfalse, this is all fsynced to
disk before the 201 status code is returned.

The obvious downside to this is that, while every header is
potentially useful, many are not but they all take up space in the
file. CouchDB cannot tell which ones will be used in the future.

Each header represents a complete and consistent snapshot of the
database, which allows CouchDB to provide reads *and* writes without
any locks.

Compaction can batch many updates together, emitting only one set of
btree nodes and only one header for many changes, because, if it were
to crash, it could try again as the input data is durably held on
disk.

To your other points: CouchDB will have 'hundreds of thousands' of
file descriptors open when you have hundreds of thousands of open
databases or views, it's hard to see how it could be otherwise.
CouchDB maintains an LRU, however, and closes open databases in order
to stay below the limit. If you've increased that limit to hundreds of
thousands and your server cannot fulfil that, it's not CouchDB's
fault. I'll just note that there are several companies running large
numbers of large couchdb databases across many machines that are
coping just fine with CouchDB's current approach to data persistence
and disk consumption. There's a lot of room for improvement, but the
approach taken at present is quite defensible.

An attachment change is a special kind of document update. The
document itself is indeed written in full to the file again, but the
unchanged attachments are not, we just point at the existing ones.
Older versions of CouchDB would replicate all attachments but more
recent ones are smarter and only replicate the ones that changed.

Finally, obviously you're frustrated by some of the light-hearted
replies in this thread, I don't think any one was being deliberately
mean but conveying nuance in email is notoriously hard. I thought the
various explanations of the value of previous headers was pretty good.
The 'value of the food you ate last year' is spot on. It's of no value
at all to you *now* but was *essential* at the time.

B.

On 1 July 2011 07:37, Zdravko Gligic <zgligic@gmail.com> wrote:
> I mapped out a set of web 2.0 level questions:
>>>> a) How many of the update headers are actually useful? =A0Is it just t=
he
>>>> last successfully written one or even just a few last ones ?
>
> Jason Smith tried reducing them to my 0.2 level:
>>> Excellent point. That's just it, isn't it?
>>> How useful are the lower rungs of a ladder? How useful is the food you
>>> ate a year ago?
>
> Jens Alfke tried re-reduced them even much lower:
>> How useful was the accident insurance policy you paid for last year and =
then
>> didn=92t have an accident? :-)
>> The update header is largely there for disaster recovery.
>
> But neither one even bothered trying to answer my question of whether
> just the last updated header or perhaps the last few are ever used.
>
>> The update header is largely there for disaster recovery.
>
> I was also under an impression that the update headers (pointers to
> the root of btree) are also somehow being used for reading
> consistency. =A0If so then this might suggest that a database could be
> rolled back to some previous point in time. =A0How far back and how
> practical is another question.
>
> In any case, this whole thread is about me trying to understand what
> is happening under the hood - strictly in trying to understand for
> what CouchDB can best be used. =A0However, since we are being a bit
> flippant about it, let me give you my 0.2 impression of CouchDB.
>
> On the surface, that RESTful API is so darn appealing and mostly
> because many portals have gone back and added such RESTful API's to
> whatever gunk they had beforehand. =A0Youtube is an excellent example.
> So, the most appealing thing to me is that by using CouchDB, a
> publicly exposable API to my own application ends up already being
> baked in by CouchDB itself, without me having to do any additional
> work. It's a whole another kettle of fish, whether I actually need to
> expose such an API or even worse - what if it's detrimental to me to
> do so and whether it is even possible not to.
>
> However, much of CouchDB is like an onion, where you can not be all
> that certain what you will end up getting underneath, until you have
> carefully peeled off a layer or two and tried it yourself. =A0Case in
> point is not just the issue of disk usage but number of concurrently
> open files that can be into tens or hundreds of thousands. There is
> just something about numbers like that, that does not feel natural -
> mostly because it raises the question of how the heck did relational
> databases ever manage to do with merely dozens of tables within a
> single database.
>
> Then there is the whole issue of servers (where much of this makes
> sense because of scalability) vs clients (where very little of this
> makes any sense because there are no multi-user concurrency issues).
> On top of that, when the same issues are raised with regard to mobile
> devices, the prevalent answer is "oh, those clients only need a much
> smaller subset". =A0So, when it comes to CouchDB on client devices, one
> is left with an impression that it is not really a database for
> storing hundreds nor even tens of thousands of records but mere dozens
> or few hundreds of key/value pairs. =A0Never mind the fact that good
> number of apps need SQLite-like querying capabilities - right down to
> full text searching and table joining.
>
> But when all is said and done, while we are discussing what seem to be
> quite large disk usage inefficiencies, I can not help but keep
> thinking about those wonderful attachments capabilities. =A0What exactly
> are they for, if not for attaching "things" that themselves can be of
> appreciable size and especially if we consider multimedia and all.
> So, what happens when one of those attachment(s) change? =A0Do the old
> revs with the old attachments become dead and a new rev is written out
> with the new attachment(s). =A0Does all of this not magnify the overall
> disk "wastage" problem by the sheer size of such attachment(s).
>
> Sincerely,
> teslan
>