couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Kowalski <>
Subject Re: CouchDB got 100% slower in the past 3 weeks
Date Mon, 07 Sep 2015 09:48:01 GMT
Hi folks,

the performance issues seem to be fixed for COUCHDB-2796.

Here is how CouchDB internals look like for an document update after the patch:

The heavy bar for couch_epi_functions_gen:providers/4 in the middle
completely disappeared.

You may wonder why all other bars in the svgs are now wider. Given the
fact that we are always watching one operation to complete in a
flamegraph which splits the calls up in percentage of CPU time this is
the expected behaviour for a performance improve.

When we shrink an perfomance-heavy call in the graph from 30% to lets
say 0.5% the others fall more into account on the next graph. So in
this case the growth of all other bars other than the ones from
couch-epi is a good thing.

Same for the calls in the flamegraph for a simple GET of a document,
which previously added up to 33% CPU time for the given operation
(reading and returning a doc):

Here are the original flamegraphs from Sunday, in case you missed them:

Update doc:
Simple GET:

The PouchDB project also confirmed that the performance decrease is
fixed now, but it seems some attachment related tests are failing now.

Big thanks to everyone involved fixing this! :)


PS: in case if you are interested in an intro to flamegraphs and
profiling, this war-story [1] is quite interesting. It is not Erlang
but the concept is language agnostic and therefore easy to apply to


On Sun, Aug 30, 2015 at 11:15 PM, Robert Kowalski <> wrote:
> Hi list,
> I got pinged as our friends from PouchDB notices that their testsuite
> with CouchDB 2 as a backend suddenly takes 100% longer (20mins instead
> of 10). [1] Because the diff was so significant I got really curious
> and worried about it.
> This testsuite just took 10min
> (22 days old)
> vs
> (yesterday)
> which took like the other runs these days 20min.
> I wasn't really sure, maybe the travis VMs changed. Or Pouch.
> Based on the report I created a few flamegraphs to poke around what
> changed in CouchDBs internals. They look quite different to the ones I
> created in the past weeks:
> this flamegraphs shows the update of a doc. couch-epi takes 33% of the
> time and blocks.
> in this flamegraph I receive a document. I have 3 blocking calls to
> couch_epi, adding up to 21% time of the request.
> The report from Nolan (perf decrease of 100% in a timeframe from 3
> weeks) fits into the timeframe where we added couch_epi. As almost
> every module uses couch_epi the performance decrease of almost all
> APIs also fits into the scheme. And I think the flamegraphs show that
> the additional time is spent in couch_epi.
> - I see that couch_epi uses the codeserver internally, would it be
> possible to use faster ETS tables?
> - Anything else we could do?
> Best,
> Robert
> [1]

View raw message