couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: Faster updates, optional ACID
Date Mon, 05 Jan 2009 19:51:47 GMT

On Jan 5, 2009, at 2:32 PM, Damien Katz wrote:

>
> On Jan 5, 2009, at 1:59 PM, Geir Magnusson Jr. wrote:
>
>>
>> On Jan 5, 2009, at 12:54 PM, Damien Katz wrote:
>>
>>> It was brought to my attention that commits on OS X were very slow  
>>> with the latest releases of Erlang. After I upgraded to the most  
>>> recent version, I found them to be indeed slow, slowing the tests  
>>> down to point it was painful to run them. It appears, any disk  
>>> sync from Erlang now takes somewhere between 50 and 100ms, up from  
>>> the previous times of ~5ms. This is almost certainly due to the  
>>> F_FULLFSYNC flag the erlang file handling now uses on darwin based  
>>> systems, but it's surprising how bad performance on OS X. A little  
>>> investigation had shown other database engines have similar issues  
>>> on OS X.
>>
>> One user, though,  reported a huge performance difference between  
>> 0.8 and trunk *using the same version of Erlang*.
>>
>> http://mail-archives.apache.org/mod_mbox/couchdb-user/200901.mbox/%3C49619897.7000104@gmail.com%3E
>>
>> To me, this hints that something changed in CouchDB that triggers  
>> usage of fcntl(F_FULLFSYNC).
>>
>> I haven't tried to duplicate but will.  If this can be verified, I  
>> think that this is an important mystery to solve.
>>
>>>
>>>
>>> To address this problem, I implemented delayed commit  
>>> functionality. We had always intended to implement delayed commit  
>>> for performance reasons, but hadn't had the need until now. This  
>>> makes updates much faster in the general case, but with the caveat  
>>> they aren't flushed completely to disk right way. If you can't  
>>> tolerate the possible loss of recent updates, you can use the  
>>> "full commit" option for ACID commits.
>>
>> So this bring up a question.  There is no way to get the same  
>> durability semantics on linux as you can get on OS X with F_FULLSYNC.
>
> On linux, as always it depends (on distro, file system, etc), but  
> generally fsync flushes to disk, or so I've been told by those who  
> should know and I've not seen credible evidence otherwise. But if  
> fsync is broken by default on Linux (like say Debian based distros),  
> file a bug and we'll see about get Erlang patched with the proper  
> apis (the Erlang F_FULLFSYNC change was from us too).

fsync() on Linux and OS X flushes to disk.  I'm not suggesting that it  
doesn't.

What it doesn't do is flush the write caches *on* the disk unit itself  
(which is what F_FULLSYNC supposedly does, hence the sloth)

IOW, with fsync(), there's no guarantee that the bits get written to  
the physical media.  As far as the OS knows, the FS caches are flushed  
to the device, but the device may still be holding in it's own RAM.

re the FULLFSYNC change, do you have the option to not have it used,  
but have fsync() used instead?

>
>
>> This means that the "full commit" option really gives you different  
>> levels of durability, depending on whether or not you are on OS X.
>>
>> And thinking more about what appears to be the perf bug/slowdown in  
>> CouchDB code, might his warrant three options?
>>
>> 1) delayed commit (what you did last night)
>> 2) fsync() commit (what I suspect Couch did on and around 0.8)
>> 3) optional F_FULLSYNC commit, on OS X and any other platform that  
>> provides this level of commit
>>
>
> If necessary and possible, we'll patch the Erlang VM.

That seems like a bad idea to me - I'd think you'd want to stay out of  
the VM business.

> But if a platform doesn't support proper flushing, then it's not a  
> platform that can support an ACID database.

We're not communicating well here.

"proper flushing" depends on what you want to do - if you need your  
data to in confirmed permanent storage so that it can survive a crash  
or power cut, then w/o special configuration (e.g. battery-backed  
RAID, for example), I don't think that you're going to get assurance  
on linux.

Do you see what I'm saying?


>
>
>>>
>>>
>>> For full acid commit, add a header field to the doc PUT or  
>>> _bulk_docs POST like this:
>>> X-Couch-Full-Commit:true
>>>
>>> Then couchdb will completely commit the change before returning.
>>
>>
>> That's cool but it puts the burden on the client -
>
> True, but they already have the API they must conform too, I don't  
> see this option as being particularly burdensome unless it's simply  
> the wrong default.

But it keeps adding requirements to the API that aren't really in the  
application domain necessarily.

>
>
>> why not make it a config option, so that the db admin can choose  
>> the durability level in general, and let clients that know they are  
>> talking to couch override w/ a header?
>>
>
> Definitely, I think commit options should be settable per-database.  
> But for now I was just wanting to address the slowdown, especially  
> for replication and the tests, to keep everyone productive. More  
> commit features and options is lower priority work for now, I was  
> just addresses the most serious slowdown.

That makes sense, but IMO you papered over the root problem.  It's  
good to keep people working, but I think the issue deserves a look.  I  
don't know erlang, or I would look myself.

geir

>
>
> Also, having the default be delayed commit will help us flush out  
> any problems, especially for usability and real productioning  
> testing. If it's broken, either a simple bug or by design, we need  
> to know it as soon as possible.
>
> -Damien
>
>>
>> geir
>>
>>>
>>>
>>> Also, if you have several delayed updates and you want to make  
>>> sure they all made it to disk, you can invoke POST /db/ 
>>> _ensure_full_commit and all outstanding commits are flushed to disk.
>>>
>>> The view engine has been already modified to deal with delayed  
>>> commits too, it ensures it never fully commits it's own indexes to  
>>> disk if the documents indexed aren't already committed to disk.
>>>
>>> The last remaining work item is db server crash detection, so that  
>>> clients can detect when a server has crashed and potentially lost  
>>> updates. This is pretty simple, each db server just needs a unique  
>>> ID generated at it's startup. Client retrieve this value at the  
>>> beginning of the writes and then checks that the value is the same  
>>> once down a flushed to disk. If not, we know we maybe have lost  
>>> some updates and we redo the replication from the last known good  
>>> commit.
>>>
>>> Right now the default is to delay the commits, because I think  
>>> that will be the most common use case but I'm really not sure. I  
>>> definitely want the commits delayed for the test suite, to keep  
>>> things running fast.
>>>
>>> -Damien
>>>
>>>
>>
>


Mime
View raw message