Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <4c2277bc0904270940w4f761a5bu524f20bb9a408a29@mail.gmail.com>
References: <4c2277bc0904262152h204185ccl8b4c084d3c68c37e@mail.gmail.com>
	 <EDDAAE68-2052-4A5A-96ED-D0A03DFC31E5@apache.org>
	 <4c2277bc0904270940w4f761a5bu524f20bb9a408a29@mail.gmail.com>
Date: Mon, 27 Apr 2009 10:49:45 -0700
Message-ID: <4c2277bc0904271049x1206531fg67852b85870cc4b7@mail.gmail.com>
Subject: Re: An asynchronous write bug in CouchDB 0.9?
From: Steven Hazel <sah@saucelabs.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=0015175cf82ec30be004688cf85e

--0015175cf82ec30be004688cf85e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Okay, I think I've confirmed this theory.  Thanks to your guidance I managed
to create a python script that can reproduce the problem on my macbook:
http://saucelabs.com/couch-bug.py

(To run it, you need the latest trunk version of couchdb-python.)

Looking at our logs, we started seeing this problem just after we crossed
the threshold to >100 dbs.  And very frequent reads to the problematic
database fixed the issue, because they made it always more recently active
than most of the others.

- Steve

On Mon, Apr 27, 2009 at 9:40 AM, Steven Hazel <sah@saucelabs.com> wrote:

> I think we do have a lot of active databases -- we have a view that is run
> frequently against each of our 137 databases.
>
> - Steve
>
>
> On Mon, Apr 27, 2009 at 6:20 AM, Damien Katz <damien@apache.org> wrote:
>
>> It looks like the db server instance is being restarted before it commits
>> the header, but it's odd you aren't seeing any errors on the console. If
>> it's a crash it should dump an error to the console.
>>
>> One thing that occurs to me, do you have a lot of active databases (> 100)
>> ? I think it's possible when you hit the max dbs open, CouchDB will shutdown
>> an idle instance with pending deferred commit, which is a bug.
>>
>> -Damien
>>
>>
>>
>> On Apr 27, 2009, at 12:52 AM, Steven Hazel wrote:
>>
>>  Hi,
>>>
>>> Last week we discovered that records in one of our CouchDB databases were
>>> occasionally being reverted to the previous revision. We believe this is
>>> due
>>> to a bug in CouchDB 0.9 related to asynchronous writes.  Turning off
>>> asynchronous writes with the X-Couch-Full-Commit header fixes the problem
>>> for us.  I've written up the problem in more detail here:
>>> http://saucelabs.com/couch-bug.html
>>>
>>> We haven't yet reproduced this problem outside our production
>>> environment,
>>> but can trigger it there on-demand.  Although we have a work-around that
>>> solves the immediate problem for us, we're happy to help out in tracking
>>> down the underlying issue.  Any advice on how to proceed would be
>>> appreciated.
>>>
>>> --
>>> Steven Hazel
>>> Sauce Labs
>>> Cofounder and Director, Development
>>> http://twitter.com/sahazel
>>>
>>
>>
>
>
> --
> Steven Hazel
> Sauce Labs
> Cofounder and Director, Development
> http://twitter.com/sahazel
>


-- 
Steven Hazel
Sauce Labs
Cofounder and Director, Development
http://twitter.com/sahazel

--0015175cf82ec30be004688cf85e--