couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CGS <>
Subject Re: Post-mortem
Date Sat, 12 May 2012 23:35:28 GMT
Well, my understanding for that specific point was that CouchDB was
crashing while writing the docs. That can be because of an error appeared
with high frequency or because the queue of documents to be written became
too long. The first is with very low chances if there is no attack
(remember, in that context, CouchDB crash was registered while inserting
docs in the database, not during compaction or building a view or anything
else), while the second can easily appear when the traffic is higher than
what the harddisk can digest. That made me think pretty much of the
conditions I spoke about.

If you want to reproduce the failure, just try to add single-doc requests
at a very high rate (standard behavior of a monitored gen_server with
listener in Erlang when the queued messages go beyond the predefined
length). But, that is something you cannot avoid if you work directly on
the hdd in whatever language you may develop the database (generally,
queued messages are RAM usage). This is the price CouchDB has to pay for
securing the data. So, if someone uses CouchDB, he/she should be aware of
that risk, unless either that someone doesn't read the CouchDB
documentation or that someone doesn't have any knowledge about hardware. In
either case, that someone shouldn't start speaking about such a so called
failure, don't you think? I would be embarrassed to report such a "problem"
unless I am sure it's coming from other sources then the ones I spoke
about. As far as I could notice, those guys didn't provide any detail about
what would cause that problem. Even worse, they admitted that by increasing
the speed of the harddisk, they noticed a slight improvement of the CouchDB
capabilities (of course, if you improve the hdd speed, you increase the
rate of document insertion which keeps the queue shorter unless you start
again to increase the speed of insertion). So, it's either me, or those
guys didn't understand CouchDB usage (considering they know something about
hardware). And that leads to the unfortunate conclusion that those guys
chose a product without thinking if that product fitted their project
design or not by conception. I do not say CouchDB cannot be used at high
rate document insertion, but one needs to pay attention how this can be
achieved. Also, the fact that they reported some bugs they discovered and
they got fast solutions from the devs proves that the devs are serious
about this product and for that they should deserve congratulations.

I do not want to offend those guys there because I know nothing about them,
but they should try to understand that coming with such "advertisement",
they need to be sure they have no blame there. I would write that kind of
article after I stopped being pissed off. But that is me and,
unfortunately, they acted too soon in my opinion.


On Fri, May 11, 2012 at 5:32 PM, Eli Stevens (Gmail)

> On Fri, May 11, 2012 at 7:57 AM, CGS <> wrote:
> > What I don't understand is the followings:
> > 1. Those guys wanted a single front-end server which should keep up with
> > the incoming requests, correct? As far as I understood, CouchDB
> philosophy
> > is based on safety of the data, which was implemented as direct writes on
> > harddisk. So, having only one front-end server on which you force its hdd
> > to to keep up with the high speed internet connection is just like you
> want
> > to force a river to flow only through a mouse hole.
> From my understanding of the post, the core issue wasn't a mismatch in
> scale between desired throughput (the river) and available throughput
> (the mousehole), it was that under high enough load CouchDB stopped
> honoring certain classes of requests entirely.  That's not a "too
> slow" problem, it's a "fell over and can't get up" problem.
> I think it's very important that effort is made to reproduce and
> address these issues, since without being able to put more definite
> bounds on them, *everyone* is going to wonder if their load is high
> enough to trigger the problems.  Heck, I'm wondering it, and I don't
> typically have more than a couple hundred docs per DB (but a lot of
> DBs, and hundreds of megs of attachments per DB).
> Eli

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message