Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of skippy.hammond@gmail.com
 designates 74.125.82.52 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=OPGBXY/FFIqaG1HbXjYRAZcU9Ut4I2NtgZI9qNsvwqh97GdsFb2eRIthJltULzunJY
         nv3sPcGiWjCv8MRCSF0jVbEz+BYmh6Jc+LSy2j+7XArswFSTTvuiiybMkrlkUPL9HyWz
         UkJvjRxxDRwD58VxzNWbUAT9pKvg3ocJXTUdM=
Message-ID: <4B885281.503@gmail.com>
Date: Sat, 27 Feb 2010 10:00:17 +1100
From: Mark Hammond <skippy.hammond@gmail.com>
Reply-To: mhammond@skippinet.com.au
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US;
 rv:1.9.1.8) Gecko/20100216 Lightning/1.0b1 Thunderbird/3.0.2
MIME-Version: 1.0
To: dev@couchdb.apache.org
CC: =?ISO-8859-1?Q?Juhani_R=E4nkimies?= <juhani@juranki.com>
Subject: Re: Windows 0.11 snapshot
References: <4B85CFDD.3070308@skippinet.com.au>
	 <e55356641002250922p2e307b79s4f4542ba3a677030@mail.gmail.com>
	 <4B870293.7080802@skippinet.com.au>
 <e55356641002252118v5fcd14e5nd6e8663cf73d1a4@mail.gmail.com>
In-Reply-To: <e55356641002252118v5fcd14e5nd6e8663cf73d1a4@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 26/02/2010 4:18 PM, Juhani R�nkimies wrote:
>
> The branch is an experiment, trying to find a solution for the
> compaction and db-delete problems on windows

Ack - I should have guessed - sorry about that :)

> There is one patch specific to the windows build process; for the case
> when path to inno setup contains spaces.
> http://github.com/juranki/couchdb/commit/0d5ec88f08a0519fdf9521730361c6da0c3d4cb4

I've incorporated that one.

Regarding the file versioning - I'm sorry that we haven't kept you in 
the loop a little better, but with Damien's guidance we think we have 
found a better strategy for this.

The root of this strategy comes from a realization that if a file is 
opened on windows with FILE_SHARE_DELETE, the file can be deleted *or 
renamed* while it is open.  One limitation is that a file of the same 
name can not be re-created while the old one still has handles open (the 
'deleted but still open' file still appears in directory listings until 
the handle is closed, for example)

Given this, what we can do is something like:

* Arrange for erlang to be able to open the DB and view files with this 
flag.
* Instead of deleting a file before replacing it, we first rename the 
file to a unique name (ie, based on a UUID) in a special directory.
* As couch starts up, attempt to delete any old files in this special 
directory.  In theory, no such files should exist - the OS should take 
care of actually removing any such files even if erlang crashes.

The end result of this is that things can be made to work with a lot 
less friction than the 'file versioning' scheme.  I've a patch to 
couchdb that works when used with a patch to erlang to open *all* files 
with that flag.  The next step down this path seems to be to create an 
erlang 'port driver' for disk IO and use this instead of the builtin 
erlang file objects.  We've identified a 'bfile' erlang port driver we 
may be able to fork and adapt to our needs - using our own port driver 
would also allow optimizations to the file IO for non-windows platforms 
(eg, indicate to the OS that certain files should not get OS-level 
caching, etc)

Unfortunately, none of this has been discussed anywhere formal - much of 
it was on IRC - so I apologize if this is the first you heard of it. 
But whatever strategy we wind up with for this is almost certainly not 
going to land and be tested enough to make 0.11.

Cheers,

Mark