couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Monteiro <ma...@textovirtual.com>
Subject Re: CouchDB freezes
Date Fri, 11 May 2012 14:45:06 GMT
Now the script really is attached. Promise.

On 11 May 2012 15:43, Marco Monteiro <marco@textovirtual.com> wrote:

> I was trying nodeload but could not generate the load I need to trigger
> the problem. I attached the script. Can you tell me how to change the
> script
> to get to the load I need to trigger the problem?
>
> The attached script was making about 150 request per minute.
>
> Thanks,
> Marco
>
>
> On 11 May 2012 14:26, Robert Newson <rnewson@apache.org> wrote:
>
>> Can you reproduce this behavior with other benchmarking tools? ab,
>> nodeload, etc?
>>
>> B.
>>
>> On 11 May 2012 14:18, Marco Monteiro <marco@textovirtual.com> wrote:
>> > Each node.js process had multiple concurrent requests. I just tried with
>> > sequential requests and the problem persists.
>> >
>> > So, now I have 8 node.js processes each sending one write request only
>> > after the previous when is done. And the problem remains.
>> >
>> > The machine is not under any kind of  huge load. Both top and iostat
>> report
>> > less than 10% machine use. The machines have 8 Core Xeon with 4
>> > 10000 rpm hard disks in raid 10 and 16 Gb.of RAM.
>> >
>> > Note that I'm testing with less than 500 requests per second, at the
>> > moment.
>> >
>> > One more thing: when the problem happens, it's not that the database
>> becomes
>> > slow. It just drops the requests. And reads also fail. For example,
>> trying
>> > to
>> > use Futon I get a "connection was reset" message from firefox.
>> >
>> > This is on CouchDB 1.2. I'm going to try 1.1.1 next.
>> >
>> > Thanks,
>> > Marco
>> >
>> > On 11 May 2012 13:56, Robert Newson <rnewson@apache.org> wrote:
>> >
>> >> Perhaps CouchDB on this particular hardware isn't fast enough to cope
>> >> with 4,000 writes per second?
>> >>
>> >> Does your node.js test send every update asynchronously or is it
>> >> carefully controlling qps? For what it's worth, I've benchmarked
>> >> successfully using a node.js library called nodeload
>> >> (https://github.com/benschmaus/nodeload). It's been a while since I
>> >> last used it, and node has changed a few dozen times since then, but
>> >> it was pretty solid and sane when I was using it.
>> >>
>> >> B.
>> >>
>> >> On 11 May 2012 13:48, Marco Monteiro <marco@textovirtual.com> wrote:
>> >> > Thanks, Robert.
>> >> >
>> >> > Disabling delayed commits did make the problem start later, but it
is
>> >> still
>> >> > there.
>> >> >
>> >> > It's funny that the first think that I checked when I first saw this
>> >> > problem was to
>> >> > make sure that delayed commits where enabled.
>> >> >
>> >> > Thanks,
>> >> > Marco
>> >> >
>> >> > On 11 May 2012 13:20, Robert Newson <rnewson@apache.org> wrote:
>> >> >
>> >> >> The first thing is to ensure you have disabled delayed commits;
>> >> >>
>> >> >> curl -XPUT -d '"false"
>> localhost:5984/_config/couchdb/delayed_commits
>> >> >>
>> >> >> This is the production setting anyway (though not the default
>> because
>> >> >> of complaints from incompetent benchmarkers). This will ensure
an
>> >> >> fsync for each write and, as a consequence, will greatly smooth
your
>> >> >> insert performance. Since you said you were inserting concurrently,
>> >> >> you should not experience a slowdown either.
>> >> >>
>> >> >> B.
>> >> >>
>> >> >> On 11 May 2012 02:42, Marco Monteiro <marco@textovirtual.com>
>> wrote:
>> >> >> > Hello!
>> >> >> >
>> >> >> > I'm running a load test on CouchDB. I have a cluster of 8
node.js
>> >> servers
>> >> >> > writing to
>> >> >> > CouchDB. They write about 30000 documents per minute (500
per
>> second).
>> >> >> > There are
>> >> >> > multiple concurrent requests form each server. There are no
>> updates:
>> >> >> > documents are
>> >> >> > created and not modified.
>> >> >> >
>> >> >> > I first tried CouchDB 1.1.1 from Debian 6.4 apt repo. After
a few
>> >> >> minutes,
>> >> >> > CouchDB
>> >> >> > starts freezing for a period of 1 to 3 seconds about every
10
>> >> seconds. It
>> >> >> > keeps this
>> >> >> > behaviour for some time and eventually it starts freezing
more
>> >> frequently
>> >> >> > and for longer
>> >> >> > periods. When the database has about 1.5 million documents,
>> couchdb is
>> >> >> > freezing for
>> >> >> > more than 5 seconds each time.
>> >> >> >
>> >> >> > I then tried CouchDB 1.2, from build-couch. The freezes happen
>> with it
>> >> >> > also, but the
>> >> >> > behavior is much worse: in less than one minute it's freezing
for
>> 5
>> >> >> seconds
>> >> >> > or more,
>> >> >> > and it spends more time not doing anything than working.
>> >> >> >
>> >> >> > When testing with 1.1.1 I was writing only to one database.
With
>> 1.2 I
>> >> >> > tried with one database
>> >> >> > and then with multiple databases but the problem was exactly
the
>> same.
>> >> >> >
>> >> >> > The documents have about 10 properties, only numbers or string
>> and the
>> >> >> > strings are small
>> >> >> > (about 20 chars each). The document IDs are generated in the
app
>> and
>> >> have
>> >> >> > the format
>> >> >> >
>> >> >> >  <milliseconds since epoch>-<random 16 chars string>
>> >> >> >
>> >> >> > When CouchDB freezes, it's processor use (from top) goes to
zero.
>> It
>> >> does
>> >> >> > not reply to read or write
>> >> >> > requests. The disk does not seem to be the  problem as iostat
>> reports
>> >> >> near
>> >> >> > 0% utilization.
>> >> >> > CPU is mostly idle, and from the 16 GB of RAM, some of it
is free
>> and
>> >> is
>> >> >> > not even used to
>> >> >> > cache disk.
>> >> >> >
>> >> >> > There are no error message in Couchdb log.
>> >> >> >
>> >> >> > I tried this in two different machines and the problem is
the
>> same in
>> >> >> both.
>> >> >> >
>> >> >> > I did not change anything in the configuration files expect
>> changing
>> >> the
>> >> >> > database dir to use
>> >> >> > a RAID partition.
>> >> >> >
>> >> >> > Anyone has any idea of what the problem could be?
>> >> >> >
>> >> >> > Any help solving this issue is greatly appreciated.
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Marco
>> >> >>
>> >>
>>
>
>

Mime
View raw message