incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <randall.le...@gmail.com>
Subject Re: Couchdb crashes on Windows
Date Thu, 30 Sep 2010 06:37:48 GMT
Oh, no! I suspected the change to append to be the issue looking into this
yesterday, too. I looked in the windows file driver (R13B05) and found the
seek function (efile_seek or something??), but thought I remembered it
properly taking 64bit values and using SetFilePointerEx. I also recall
finding the append path calling the this function, but maybe I made that
part up. I didn't get a chance to fully stamp this out either, but I think
we're on the right track.
> On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams <cliffywills@aol.com>
wrote:
>>  Dave,
>>
>> I hope you are well
>>
>> I too began to look at this a couple of nights ago.
>>
>> The process that I was going to follow;
>>    Establish a base case
>>    Write some file IO tests
>>    Apply these tests to the Couchdb environment and see what differences
>> popped out.
>>
>> At the very least I thought that I could produce some tests to ascertain
why
>> 1.01 is misbehaving on windows, if Erlang is problematic  and to use as
>> future tests for new releases of Couchdb on windows.
>>
>> I downloaded the latest full Erlang windows binary as my base case
>> (currently R14B) and wrote the following very simple bit of code as my
first
>> test
>>
>> <code>
>> -module(filetest).
>> -export([main/0]).
>> main() ->
>> {ok, Binary}=file:read_file("blanks"),
>> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
>>
>> loop(1000, WriteDescr,Binary),
>> file:close(WriteDescr).
>>
>> loop(0,_NotNeeded,_NotNeeded) -> ok;
>>
>> loop(N,WriteDescr,Binary) ->
>> file:write(WriteDescr,Binary),
>> loop(N-1,WriteDescr,Binary).
>> </code>
>>
>> Note blanks is a zero filled file of about 2.2MB that i originally used
to
>> confirm that i had the same issues as Peter first reported.
>>
>> Run with a loop of 10 and you get a twenty MB file ........... nice
>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>> delete filetest.txt
>> Run again with a loop of 2000 and you get a file in excess of 4GB
..........
>> nice .......... but slow
>> delete filetest.txt
>> Run again with a loop of 4000 and you get a file in excess of 8GB
>> ...........nice............but very slow
>>
>> Now everything seems fine and dandy but .......
>>
>> Delete filetest.txt
>> Run with a loop of 10 and you get a twenty MB file ........... nice
>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>> Run again with a loop of 2000 and you get a file in excess of 4GB
..........
>> nice .......... but slow
>> Run again with a loop of 2000 and the program begins to "append" from the
>> beginning of the file so the file does not grow but instead gets
>> overwritten. This behaviour seems to occur at the 4GB file size on change
of
>> file descriptor ............. not nice ....... not good.
>>
>> I've had a look at the way Erlang is compiled for windows and there are a
>> couple of flags pertaining to large file support. Over the next few days
I
>> will attempt my own compilation of Erlang with these flags set and report
>> back. (Need to get hold of VC++ first)
>>
>> regards
>>
>> Cliff
>>
>>
>
> Excellent work!!
> And 0.11 works because it doesn't open files in append mode.
>
> The file driver uses SetFilePointer in append mode to write to eof.
>
> DWORD WINAPI SetFilePointer(
> __in HANDLE hFile,
> __in LONG lDistanceToMove,
> __inout_opt PLONG lpDistanceToMoveHigh,
> __in DWORD dwMoveMethod
> );
>
> MS documentation states that "To work with file pointers that are
> larger than a single LONG value, it is easier to use the
> SetFilePointerEx function."
> (http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
> and the driver passes NULL to lpDistanceToMoveHigh
> Suspicious.
>
> I'll digg a little here.
>
> -juhani

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message