couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <>
Subject Re: document insert rate
Date Sun, 04 Jan 2009 21:11:58 GMT

On Jan 4, 2009, at 2:59 PM, Damien Katz wrote:

> What you say is possible, but doesn't seem too likely as I don't  
> ever recall any report of a corrupt db on Linux, but before the  
> F_FILESYNC change in Erlang we would see reports of corruptions on  
> OS X due to power loss. Since the change, we've had no reports of  
> corrupt databases on OS X or Linux.

Have you seen the correlation between the # of pirates and global  
warming? :)

There may be correlation here between the Erlang update and  
corruption, but I don't see the causality -  fsync() on linux should  
only get as far as the on-device buffering if it does write buffering,  
whereas if the OS X docs are truthful, the F_FILESYNC fcntl() pushs  
the bits to the rust on the platter.  Maybe they fixed something else  
in that erlang update, or maybe the universe is smiling upon CouchDB  
users or ....

> I don't think anyone else sees update performance that slow (I'm  
> assuming these are fairly small documents?), I'd guess there is  
> something particular about your setups. Since it involves update  
> speed it wouldn't be the javascript engine. It's possible it's the  
> openssl interface, I've seen instances of wacked openssl installs  
> that slowed down Erlang but didn't crash it.

I have an idea.... try it. I bet you can't get better than that (6-8   
inserts per second) on Mac OS X unless you have stupid-fast disks  
(like maybe an SSD).

Even get a "back of the envelope" approx using time and curl - that  
came in at the same rate as Ruby, JS and Java test programs.

Everyone who has tried my experiment on IRC has reported the same  
numbers.  I can get 6/sec on a laptop (2.5GHz core2 dual core MBP w/  
4GB ram and a 7200rpm disk), and 8/sec on my desktop (a quad-core, 8  
GB w/ raided 7200rpm disks, IIRC)


> -Damien
> On Jan 4, 2009, at 1:35 PM, Geir Magnusson Jr. wrote:
>> I'm suspicious.
>> On an m1.small instance running 32-bit ubuntu 8 on ec2 running  
>> trunk, I can get 85 inserts per second with the client running on  
>> same slice, so no network hop.  it's a single core instance, but  
>> I'm not sure that would matter too much (i'll try later).
>> Now, this makes me wonder... the fsync() in linux isn't doing what  
>> F_FILESYNC does in OS X presumably....  in which case what you  
>> think is being done for you to work around the lack of an "off  
>> switch"  in the erlang VM isn't actually being done, and since no  
>> one seems to have noticed, does it really matter? :)
>> To be fair, I'm going to try a better experiment on linux w/ more  
>> cores and more clients...
>> geir
>> On Jan 4, 2009, at 12:45 PM, Jan Lehnardt wrote:
>>> On 4 Jan 2009, at 14:09, Geir Magnusson Jr. wrote:
>>>> 2) Is anyone using CouchDB in a manner that really requires this  
>>>> level of data security?  I appreciate having the *option* to turn  
>>>> on a mode like this, but I don't think I need it all the time.  I  
>>>> can use RAID systems that give me battery-backed cache, or I can  
>>>> make the design decision that I am happy to lose X seconds of  
>>>> data in a tradeoff (e.g. do the deep fnctl() every X seconds.....)
>>> Everybody. CouchDB doesn't have an `off`-switch. You just terminate
>>> the Erlang VM. Without this design, this wouldn't be possible. There
>>> has been no discussion about optionally making different trade-offs
>>> for speed and against data integrity, but this is not off-the-map.
>>> Cheers
>>> Jan
>>> --

View raw message