couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: silent view index file corruption
Date Tue, 07 Sep 2010 19:45:04 GMT
On Apr 12, 2010, at 8:12 PM, Adam Kocoloski wrote:

> Back to the corruption ... after I spent some time reviewing CouchDB's fsync settings
I'm wondering why we don't fsync the view index file before writing a header.  I understand
why we don't do it after the header write -- we can always just reindex the last few updates
-- but the pre-sync would preserve some write ordering and possibly prevent some sources of
corruption.
> 
> Adam

Dragging up this thread again ... I encountered another situation (again on EC2) where multiple
.view files were not updating.  I discovered that in each case the index_header for the file
claimed that the root of the ID tree was at a location where I found nothing but a bunch of
zeroes.  These instances were running XFS, so the zeroes indicate to me that the btree nodes
were not written to disk.  I was able to recover the indexes by chopping off the last ~20k
of each file, thus causing CouchDB to use an older header for the file which pointed to durably
stored btree nodes.

In theory the fsync before header write would eliminate this situation.  I'm going to add
it unless there are objections.  I'll make it configurable by the "before_header" config setting
that controls the sync behavior of databases.

I think it would also be nice to have a little command-line tool to look for the last few
index_headers and offer to truncate the file so that one of these is used instead of the latest
one.

Adam


Mime
View raw message