couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Monteiro <ma...@textovirtual.com>
Subject Re: CouchDB freezes
Date Fri, 11 May 2012 13:18:29 GMT
Each node.js process had multiple concurrent requests. I just tried with
sequential requests and the problem persists.

So, now I have 8 node.js processes each sending one write request only
after the previous when is done. And the problem remains.

The machine is not under any kind of  huge load. Both top and iostat report
less than 10% machine use. The machines have 8 Core Xeon with 4
10000 rpm hard disks in raid 10 and 16 Gb.of RAM.

Note that I'm testing with less than 500 requests per second, at the
moment.

One more thing: when the problem happens, it's not that the database becomes
slow. It just drops the requests. And reads also fail. For example, trying
to
use Futon I get a "connection was reset" message from firefox.

This is on CouchDB 1.2. I'm going to try 1.1.1 next.

Thanks,
Marco

On 11 May 2012 13:56, Robert Newson <rnewson@apache.org> wrote:

> Perhaps CouchDB on this particular hardware isn't fast enough to cope
> with 4,000 writes per second?
>
> Does your node.js test send every update asynchronously or is it
> carefully controlling qps? For what it's worth, I've benchmarked
> successfully using a node.js library called nodeload
> (https://github.com/benschmaus/nodeload). It's been a while since I
> last used it, and node has changed a few dozen times since then, but
> it was pretty solid and sane when I was using it.
>
> B.
>
> On 11 May 2012 13:48, Marco Monteiro <marco@textovirtual.com> wrote:
> > Thanks, Robert.
> >
> > Disabling delayed commits did make the problem start later, but it is
> still
> > there.
> >
> > It's funny that the first think that I checked when I first saw this
> > problem was to
> > make sure that delayed commits where enabled.
> >
> > Thanks,
> > Marco
> >
> > On 11 May 2012 13:20, Robert Newson <rnewson@apache.org> wrote:
> >
> >> The first thing is to ensure you have disabled delayed commits;
> >>
> >> curl -XPUT -d '"false" localhost:5984/_config/couchdb/delayed_commits
> >>
> >> This is the production setting anyway (though not the default because
> >> of complaints from incompetent benchmarkers). This will ensure an
> >> fsync for each write and, as a consequence, will greatly smooth your
> >> insert performance. Since you said you were inserting concurrently,
> >> you should not experience a slowdown either.
> >>
> >> B.
> >>
> >> On 11 May 2012 02:42, Marco Monteiro <marco@textovirtual.com> wrote:
> >> > Hello!
> >> >
> >> > I'm running a load test on CouchDB. I have a cluster of 8 node.js
> servers
> >> > writing to
> >> > CouchDB. They write about 30000 documents per minute (500 per second).
> >> > There are
> >> > multiple concurrent requests form each server. There are no updates:
> >> > documents are
> >> > created and not modified.
> >> >
> >> > I first tried CouchDB 1.1.1 from Debian 6.4 apt repo. After a few
> >> minutes,
> >> > CouchDB
> >> > starts freezing for a period of 1 to 3 seconds about every 10
> seconds. It
> >> > keeps this
> >> > behaviour for some time and eventually it starts freezing more
> frequently
> >> > and for longer
> >> > periods. When the database has about 1.5 million documents, couchdb is
> >> > freezing for
> >> > more than 5 seconds each time.
> >> >
> >> > I then tried CouchDB 1.2, from build-couch. The freezes happen with it
> >> > also, but the
> >> > behavior is much worse: in less than one minute it's freezing for 5
> >> seconds
> >> > or more,
> >> > and it spends more time not doing anything than working.
> >> >
> >> > When testing with 1.1.1 I was writing only to one database. With 1.2 I
> >> > tried with one database
> >> > and then with multiple databases but the problem was exactly the same.
> >> >
> >> > The documents have about 10 properties, only numbers or string and the
> >> > strings are small
> >> > (about 20 chars each). The document IDs are generated in the app and
> have
> >> > the format
> >> >
> >> >  <milliseconds since epoch>-<random 16 chars string>
> >> >
> >> > When CouchDB freezes, it's processor use (from top) goes to zero. It
> does
> >> > not reply to read or write
> >> > requests. The disk does not seem to be the  problem as iostat reports
> >> near
> >> > 0% utilization.
> >> > CPU is mostly idle, and from the 16 GB of RAM, some of it is free and
> is
> >> > not even used to
> >> > cache disk.
> >> >
> >> > There are no error message in Couchdb log.
> >> >
> >> > I tried this in two different machines and the problem is the same in
> >> both.
> >> >
> >> > I did not change anything in the configuration files expect changing
> the
> >> > database dir to use
> >> > a RAID partition.
> >> >
> >> > Anyone has any idea of what the problem could be?
> >> >
> >> > Any help solving this issue is greatly appreciated.
> >> >
> >> > Thanks,
> >> > Marco
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message