couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Ränkimies <juh...@juranki.com>
Subject Re: Couchdb crashes on Windows
Date Thu, 30 Sep 2010 05:18:12 GMT
On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams <cliffywills@aol.com> wrote:
>  Dave,
>
> I hope you are well
>
> I too began to look at this a couple of nights ago.
>
> The process that I was going to follow;
>    Establish a base case
>    Write some file IO tests
>    Apply these tests to the Couchdb environment and see what differences
> popped out.
>
> At the very least I thought that I could produce some tests to ascertain why
> 1.01 is misbehaving on windows, if Erlang is problematic  and to use as
> future tests for new releases of Couchdb on windows.
>
> I downloaded the latest full Erlang windows binary as my base case
> (currently R14B) and wrote the following very simple bit of code as my first
> test
>
> <code>
> -module(filetest).
> -export([main/0]).
> main() ->
> {ok, Binary}=file:read_file("blanks"),
> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
>
> loop(1000, WriteDescr,Binary),
> file:close(WriteDescr).
>
> loop(0,_NotNeeded,_NotNeeded) -> ok;
>
> loop(N,WriteDescr,Binary) ->
> file:write(WriteDescr,Binary),
> loop(N-1,WriteDescr,Binary).
> </code>
>
> Note blanks is a zero filled file of about 2.2MB that i originally used to
> confirm that i had the same issues as Peter first reported.
>
> Run with a loop of 10 and you get a twenty MB file ........... nice
> Run again with a loop of 10 and you get a Forty MB file ......... nice
> delete filetest.txt
> Run again with a loop of 2000 and you get a file in excess of 4GB ..........
> nice .......... but slow
> delete filetest.txt
> Run again with a loop of 4000 and you get a file in excess of 8GB
> ...........nice............but very slow
>
> Now everything seems fine and dandy but .......
>
> Delete filetest.txt
> Run with a loop of 10 and you get a twenty MB file ........... nice
> Run again with a loop of 10 and you get a Forty MB file ......... nice
> Run again with a loop of 2000 and you get a file in excess of 4GB ..........
> nice .......... but slow
> Run again with a loop of 2000 and the program begins to "append" from the
> beginning of the file so the file does not grow but instead gets
> overwritten. This behaviour seems to occur at the 4GB file size on change of
> file descriptor ............. not nice ....... not good.
>
> I've had a look at the way Erlang is compiled for windows and there are a
> couple of flags pertaining to large file support. Over the next few days I
> will attempt my own compilation of Erlang with these flags set and report
> back. (Need to get hold of VC++ first)
>
> regards
>
> Cliff
>
>

Excellent work!!
And 0.11 works because it doesn't open files in append mode.

The file driver uses SetFilePointer in append mode to write to eof.

DWORD WINAPI SetFilePointer(
  __in         HANDLE hFile,
  __in         LONG lDistanceToMove,
  __inout_opt  PLONG lpDistanceToMoveHigh,
  __in         DWORD dwMoveMethod
);

MS documentation states that "To work with file pointers that are
larger than a single LONG value, it is easier to use the
SetFilePointerEx function."
(http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
and the driver passes NULL to lpDistanceToMoveHigh
Suspicious.

I'll digg a little here.

-juhani

Mime
View raw message