incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Monteiro <ma...@textovirtual.com>
Subject Re: CouchDB freezes
Date Fri, 11 May 2012 14:43:36 GMT
I was trying nodeload but could not generate the load I need to trigger
the problem. I attached the script. Can you tell me how to change the script
to get to the load I need to trigger the problem?

The attached script was making about 150 request per minute.

Thanks,
Marco

On 11 May 2012 14:26, Robert Newson <rnewson@apache.org> wrote:

> Can you reproduce this behavior with other benchmarking tools? ab,
> nodeload, etc?
>
> B.
>
> On 11 May 2012 14:18, Marco Monteiro <marco@textovirtual.com> wrote:
> > Each node.js process had multiple concurrent requests. I just tried with
> > sequential requests and the problem persists.
> >
> > So, now I have 8 node.js processes each sending one write request only
> > after the previous when is done. And the problem remains.
> >
> > The machine is not under any kind of  huge load. Both top and iostat
> report
> > less than 10% machine use. The machines have 8 Core Xeon with 4
> > 10000 rpm hard disks in raid 10 and 16 Gb.of RAM.
> >
> > Note that I'm testing with less than 500 requests per second, at the
> > moment.
> >
> > One more thing: when the problem happens, it's not that the database
> becomes
> > slow. It just drops the requests. And reads also fail. For example,
> trying
> > to
> > use Futon I get a "connection was reset" message from firefox.
> >
> > This is on CouchDB 1.2. I'm going to try 1.1.1 next.
> >
> > Thanks,
> > Marco
> >
> > On 11 May 2012 13:56, Robert Newson <rnewson@apache.org> wrote:
> >
> >> Perhaps CouchDB on this particular hardware isn't fast enough to cope
> >> with 4,000 writes per second?
> >>
> >> Does your node.js test send every update asynchronously or is it
> >> carefully controlling qps? For what it's worth, I've benchmarked
> >> successfully using a node.js library called nodeload
> >> (https://github.com/benschmaus/nodeload). It's been a while since I
> >> last used it, and node has changed a few dozen times since then, but
> >> it was pretty solid and sane when I was using it.
> >>
> >> B.
> >>
> >> On 11 May 2012 13:48, Marco Monteiro <marco@textovirtual.com> wrote:
> >> > Thanks, Robert.
> >> >
> >> > Disabling delayed commits did make the problem start later, but it is
> >> still
> >> > there.
> >> >
> >> > It's funny that the first think that I checked when I first saw this
> >> > problem was to
> >> > make sure that delayed commits where enabled.
> >> >
> >> > Thanks,
> >> > Marco
> >> >
> >> > On 11 May 2012 13:20, Robert Newson <rnewson@apache.org> wrote:
> >> >
> >> >> The first thing is to ensure you have disabled delayed commits;
> >> >>
> >> >> curl -XPUT -d '"false" localhost:5984/_config/couchdb/delayed_commits
> >> >>
> >> >> This is the production setting anyway (though not the default because
> >> >> of complaints from incompetent benchmarkers). This will ensure an
> >> >> fsync for each write and, as a consequence, will greatly smooth your
> >> >> insert performance. Since you said you were inserting concurrently,
> >> >> you should not experience a slowdown either.
> >> >>
> >> >> B.
> >> >>
> >> >> On 11 May 2012 02:42, Marco Monteiro <marco@textovirtual.com>
wrote:
> >> >> > Hello!
> >> >> >
> >> >> > I'm running a load test on CouchDB. I have a cluster of 8 node.js
> >> servers
> >> >> > writing to
> >> >> > CouchDB. They write about 30000 documents per minute (500 per
> second).
> >> >> > There are
> >> >> > multiple concurrent requests form each server. There are no
> updates:
> >> >> > documents are
> >> >> > created and not modified.
> >> >> >
> >> >> > I first tried CouchDB 1.1.1 from Debian 6.4 apt repo. After a
few
> >> >> minutes,
> >> >> > CouchDB
> >> >> > starts freezing for a period of 1 to 3 seconds about every 10
> >> seconds. It
> >> >> > keeps this
> >> >> > behaviour for some time and eventually it starts freezing more
> >> frequently
> >> >> > and for longer
> >> >> > periods. When the database has about 1.5 million documents,
> couchdb is
> >> >> > freezing for
> >> >> > more than 5 seconds each time.
> >> >> >
> >> >> > I then tried CouchDB 1.2, from build-couch. The freezes happen
> with it
> >> >> > also, but the
> >> >> > behavior is much worse: in less than one minute it's freezing
for 5
> >> >> seconds
> >> >> > or more,
> >> >> > and it spends more time not doing anything than working.
> >> >> >
> >> >> > When testing with 1.1.1 I was writing only to one database. With
> 1.2 I
> >> >> > tried with one database
> >> >> > and then with multiple databases but the problem was exactly the
> same.
> >> >> >
> >> >> > The documents have about 10 properties, only numbers or string
and
> the
> >> >> > strings are small
> >> >> > (about 20 chars each). The document IDs are generated in the app
> and
> >> have
> >> >> > the format
> >> >> >
> >> >> >  <milliseconds since epoch>-<random 16 chars string>
> >> >> >
> >> >> > When CouchDB freezes, it's processor use (from top) goes to zero.
> It
> >> does
> >> >> > not reply to read or write
> >> >> > requests. The disk does not seem to be the  problem as iostat
> reports
> >> >> near
> >> >> > 0% utilization.
> >> >> > CPU is mostly idle, and from the 16 GB of RAM, some of it is free
> and
> >> is
> >> >> > not even used to
> >> >> > cache disk.
> >> >> >
> >> >> > There are no error message in Couchdb log.
> >> >> >
> >> >> > I tried this in two different machines and the problem is the
same
> in
> >> >> both.
> >> >> >
> >> >> > I did not change anything in the configuration files expect
> changing
> >> the
> >> >> > database dir to use
> >> >> > a RAID partition.
> >> >> >
> >> >> > Anyone has any idea of what the problem could be?
> >> >> >
> >> >> > Any help solving this issue is greatly appreciated.
> >> >> >
> >> >> > Thanks,
> >> >> > Marco
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message