incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject document insert rate
Date Sun, 04 Jan 2009 13:09:45 GMT
As some know from my pestering on irc, I've been trying to figure out  
how fast CouchDB is for inserts.

I've tried using programs in JS, Ruby and Java, and a script using  
curl, and each time I can't do better than 6-8 inserts / sec.  I get  
about 6/sec on my laptop, and 8/sec on my 8GB, 8 core desktop mac ( so  
I don't think I haven't thrown enough hardware at the problem...)

At first I thought I was an idiot, and was doing something wrong.  But  
now I'm not so sure.

On IRC chat, it was mentioned that erlang 5.6.5 fixed some fsync()  
problem, so now it does a full sync to be sure data is safe.  Ok (more  
on this in a moment).

I tried to duplicate this rate using a simple C program, and compare  
fsync(), which only flushes to devices, with fcntl(F_FULLFSYNC), which  
is supposed to get the storage devices to flush their own buffers and  
do a physical write.

In a loop, I'd do a fprintf() with a 100 or so byte string, and then  
either a fsync() or a fcntl() right after.

I'd loop 1000 times.

For fsync(), the loop time was trivial - on the order of 50k usec.

For fcntl(), the loop time was painful - between 3 and 4 seconds.

So the "deep fsync()" is slow - but I was still getting > 250 / sec.

I realize that my trivial program isn't doing everything that Couch  
might be doing, but still - it's 2 orders of magnitude faster.

So the questions are :

1) What exactly is CouchDB doing to get only 6 inserts per second?

2) Is anyone using CouchDB in a manner that really requires this level  
of data security?  I appreciate having the *option* to turn on a mode  
like this, but I don't think I need it all the time.  I can use RAID  
systems that give me battery-backed cache, or I can make the design  
decision that I am happy to lose X seconds of data in a tradeoff (e.g.  
do the deep fnctl() every X seconds.....)

Has this been discussed anywhere?  Always happy to catch up on a mail  
archive...

geir





Mime
View raw message