Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-Id: <78F57100-1438-4D08-B58D-1D81AE6DD10A@apache.org>
From: Damien Katz <damien@apache.org>
To: dev@couchdb.apache.org
In-Reply-To: <CC0FAF04-CF82-4F55-B180-B458D51D7466@mooseyard.com>
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Detailed info on the B-tree store? Native implementations
  thereof?
Date: Tue, 11 Aug 2009 20:03:53 -0400
References: <5B24687D-DB8B-4FB4-B738-D3A19C426846@mooseyard.com>
 <e282921e0908111037j490d361l813f5614c610cdfc@mail.gmail.com>
 <CC0FAF04-CF82-4F55-B180-B458D51D7466@mooseyard.com>


On Aug 11, 2009, at 3:07 PM, Jens Alfke wrote:

>
> On Aug 11, 2009, at 10:37 AM, Chris Anderson wrote:
>
>> Since this article, we've changed the header handling, so that we
>> don't keep it at the top of the file, but instead append the header =20=

>> at
>> the end of the file at every commit. The strict append-only nature of
>> the storage engine is the source of it's robustness. Even an extreme
>> action, like truncating the file, will not result in an inconsistent
>> state.
>
> Interesting. Does this really guarantee file integrity even in the =20
> case of power failure? (I have some experience dealing with file =20
> corruption, from working on Mac OS X components that use sqlite.)

CouchDB is completely protected from improper shutdowns (so long as =20
the filesystem doesn't corrupt already fsync'd data), even if the file =20=

gets truncated, a very common occurrence when you run out of disk space.

> The worst problem is that the disk controller will reorder sector =20
> writes to reduce seek time, which in effect means that if power is =20
> lost, some random subset of the last writes may not happen. So you =20
> won't just end up with a truncated file =97 you could have a file that =
=20
> seems intact and has a correct header at the end, but has 4k bytes =20
> of garbage somewhere within the last transaction. Does CouchDB's =20
> file structure guard against that?

First we fsync all the data and indexes, then we write and fsync the =20
headers in a separate step.

-Damien

>
> My concern with HTML5 local storage is that it's going to be used =20
> for important user data that cannot be lost, just the way native =20
> apps put irreplacable data in local files. But the data stores being =20=

> used to implement local storage are much less resilient than the =20
> filesystem itself. My experience with sqlite is that heavily-used =20
> databases on consumer machines get corrupted and lost every few =20
> months.( This isn't directly related to CouchDB itself; but it's why =20=

> I'm interested in the fault-tolerant data store it uses.)
>
>> The other aspect our API that web storage will need to be
>> concurrency-friendly is MVCC. Without MVCC you end up needing long
>> transactions between page-loads, like localStorage currently has,
>> which makes it useless for sharing state between windows.
>
> I'm still not 100% convinced by your analysis in that blog post. A =20
> script running in a web page will implicitly acquire a lock when it =20=

> accesses local storage, and release the lock at the end of the =20
> current event that it's handling (i.e. a user action or XHR =20
> response.) This is sufficiently fine-grained as to not pose a =20
> problem, I think.
>
> But Jeremy Orlow pointed out a more problematic case to me =97 the =20
> HML5 worker-thread API. Worker threads should be able to access =20
> local storage, and they don't have an event-based model; so a worker =20=

> thread will probably be within some internal 'while' loop during its =20=

> entire lifespan. There is thus no way to automatically handle =20
> transactions for it, so it will have to manually acquire and release =20=

> locks. That means that a buggy or blocked worker thread could starve =20=

> web pages in the same domain from accessing local storage. That's bad.
>
>> Maybe the easiest thing would be to just start bundling CouchDB with
>> your browser. :)
>
> In a lot of ways that would be really awesome. However, it would =20
> have a terrible effect on the download size of the browser, which is =20=

> an important consideration. (IIRC, the all-in-one double-clickable =20
> Mac CouchDB package is something like 15MB.)
>
> I like the idea, which I think you proposed, of putting a basic b-=20
> tree API into the browser, and being able to implement a lite =20
> storage system compatible with CouchDB on top of it in JS.
>
> =97Jens