incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: Data loss in CouchDB 1.0.1
Date Fri, 03 Jun 2011 14:07:02 GMT
On Fri, Jun 3, 2011 at 9:54 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
> On Jun 3, 2011, at 9:21 AM, René Brüntrup wrote:
>
>> Dear CouchDB Community,
>>
>> After restarting our productive Ubuntu 10.10 server running the official
>> Ubuntu CouchDB 1.0.1 package we observed a data loss in our database.
>>
>> Before restarting the server the application using the database worked
>> without any problems. After restarting the server some recent
>> modifications were missing from the system and we started investigating
>> the problems:
>> The last change time of the CouchDB database file dated back about two
>> months. A hex dump of the CouchDB file showed no changes after this
>> point of time. Regular database backups that where created by
>> replicating into another database contained more recent data, just the
>> main database seemed to have stopped recording changes to disc after
>> 2011-08-03. All database files are stored on the same encrypted LVM
>> volume.
>>
>> The problem sounds a lot like writer process bug in 1.0.0: All data
>> seems to be kept in RAM, but never written to disc. However, we verified
>> that Ubuntu uses the patched code and the conditions that could cause
>> the problems in version 1.0.0 do not even occur in our system.
>>
>> So far we were not able to reproduce the behavior.
>>
>> Thank you for reading and we appreciate any help.
>>
>> Kind Regards
>> René Brüntrup
>
> Hi René, thanks for this report.  Can you clarify a couple of things for us:
>
> 1) Are you certain that you were in fact writing to the database on this server and not
the replica?  Can you share some access logs towards that end?
>
> 2) Is it possible that you've inadvertently restored the database file from a backup?
>
> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the
past two months?
>

Just a quick clarification as I had to have this explained on IRC.
Writing "underneath" the LVM volume means opening a file in a path
under the mount point before mounting the encrypted volume. The
assumption being that the LVM mount happens late in the boot sequence
so it might be possible that it was mounted after CouchDB had already
been started and opened file handles to the underlying filesystem.

> CouchDB only allows clients -- including the replicator -- to read documents that have
been written to disk (though not necessarily synced).  The bug in 1.0.0 concerned the writing
of database headers.  The database file itself was definitely being updated.
>
> We take reports of this nature quite seriously and will be watching this thread closely.
Regards,
>
> --
> Adam Kocoloski
>
>
>

Mime
View raw message