incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Ränkimies <juh...@juranki.com>
Subject Re: Couchdb crashes on Windows
Date Thu, 30 Sep 2010 08:29:11 GMT
You remember correctly. efile_seek uses SetFilePointerEx variant, but
efile_write and efile_writev call SetFilePointer when in append mode .

I searched a little and found that it's possible to have WriteFile
write at EOF without first setting the file pointer (one less system
call). (http://msdn.microsoft.com/en-us/library/aa365747%28v=VS.85%29.aspx,
at the bottom of the page)

Did a quick test  and it seems to work.
http://github.com/juranki/otp/commit/9397777751de1ef20de29a911162da2385801c52

-juhani

On Thu, Sep 30, 2010 at 9:37 AM, Randall Leeds <randall.leeds@gmail.com> wrote:
> Oh, no! I suspected the change to append to be the issue looking into this
> yesterday, too. I looked in the windows file driver (R13B05) and found the
> seek function (efile_seek or something??), but thought I remembered it
> properly taking 64bit values and using SetFilePointerEx. I also recall
> finding the append path calling the this function, but maybe I made that
> part up. I didn't get a chance to fully stamp this out either, but I think
> we're on the right track.
>> On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams <cliffywills@aol.com>
> wrote:
>>>  Dave,
>>>
>>> I hope you are well
>>>
>>> I too began to look at this a couple of nights ago.
>>>
>>> The process that I was going to follow;
>>>    Establish a base case
>>>    Write some file IO tests
>>>    Apply these tests to the Couchdb environment and see what differences
>>> popped out.
>>>
>>> At the very least I thought that I could produce some tests to ascertain
> why
>>> 1.01 is misbehaving on windows, if Erlang is problematic  and to use as
>>> future tests for new releases of Couchdb on windows.
>>>
>>> I downloaded the latest full Erlang windows binary as my base case
>>> (currently R14B) and wrote the following very simple bit of code as my
> first
>>> test
>>>
>>> <code>
>>> -module(filetest).
>>> -export([main/0]).
>>> main() ->
>>> {ok, Binary}=file:read_file("blanks"),
>>> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
>>>
>>> loop(1000, WriteDescr,Binary),
>>> file:close(WriteDescr).
>>>
>>> loop(0,_NotNeeded,_NotNeeded) -> ok;
>>>
>>> loop(N,WriteDescr,Binary) ->
>>> file:write(WriteDescr,Binary),
>>> loop(N-1,WriteDescr,Binary).
>>> </code>
>>>
>>> Note blanks is a zero filled file of about 2.2MB that i originally used
> to
>>> confirm that i had the same issues as Peter first reported.
>>>
>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>> delete filetest.txt
>>> Run again with a loop of 2000 and you get a file in excess of 4GB
> ..........
>>> nice .......... but slow
>>> delete filetest.txt
>>> Run again with a loop of 4000 and you get a file in excess of 8GB
>>> ...........nice............but very slow
>>>
>>> Now everything seems fine and dandy but .......
>>>
>>> Delete filetest.txt
>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>> Run again with a loop of 2000 and you get a file in excess of 4GB
> ..........
>>> nice .......... but slow
>>> Run again with a loop of 2000 and the program begins to "append" from the
>>> beginning of the file so the file does not grow but instead gets
>>> overwritten. This behaviour seems to occur at the 4GB file size on change
> of
>>> file descriptor ............. not nice ....... not good.
>>>
>>> I've had a look at the way Erlang is compiled for windows and there are a
>>> couple of flags pertaining to large file support. Over the next few days
> I
>>> will attempt my own compilation of Erlang with these flags set and report
>>> back. (Need to get hold of VC++ first)
>>>
>>> regards
>>>
>>> Cliff
>>>
>>>
>>
>> Excellent work!!
>> And 0.11 works because it doesn't open files in append mode.
>>
>> The file driver uses SetFilePointer in append mode to write to eof.
>>
>> DWORD WINAPI SetFilePointer(
>> __in HANDLE hFile,
>> __in LONG lDistanceToMove,
>> __inout_opt PLONG lpDistanceToMoveHigh,
>> __in DWORD dwMoveMethod
>> );
>>
>> MS documentation states that "To work with file pointers that are
>> larger than a single LONG value, it is easier to use the
>> SetFilePointerEx function."
>> (http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
>> and the driver passes NULL to lpDistanceToMoveHigh
>> Suspicious.
>>
>> I'll digg a little here.
>>
>> -juhani
>

Mime
View raw message