couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <randall.le...@gmail.com>
Subject Re: Couchdb crashes on Windows
Date Thu, 30 Sep 2010 15:03:49 GMT
Niiice work!
> You remember correctly. efile_seek uses SetFilePointerEx variant, but
> efile_write and efile_writev call SetFilePointer when in append mode .
>
> I searched a little and found that it's possible to have WriteFile
> write at EOF without first setting the file pointer (one less system
> call). (http://msdn.microsoft.com/en-us/library/aa365747%28v=VS.85%29.aspx
,
> at the bottom of the page)
>
> Did a quick test and it seems to work.
>
http://github.com/juranki/otp/commit/9397777751de1ef20de29a911162da2385801c52
>
> -juhani
>
> On Thu, Sep 30, 2010 at 9:37 AM, Randall Leeds <randall.leeds@gmail.com>
wrote:
>> Oh, no! I suspected the change to append to be the issue looking into
this
>> yesterday, too. I looked in the windows file driver (R13B05) and found
the
>> seek function (efile_seek or something??), but thought I remembered it
>> properly taking 64bit values and using SetFilePointerEx. I also recall
>> finding the append path calling the this function, but maybe I made that
>> part up. I didn't get a chance to fully stamp this out either, but I
think
>> we're on the right track.
>>> On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams <cliffywills@aol.com>
>> wrote:
>>>>  Dave,
>>>>
>>>> I hope you are well
>>>>
>>>> I too began to look at this a couple of nights ago.
>>>>
>>>> The process that I was going to follow;
>>>>    Establish a base case
>>>>    Write some file IO tests
>>>>    Apply these tests to the Couchdb environment and see what
differences
>>>> popped out.
>>>>
>>>> At the very least I thought that I could produce some tests to
ascertain
>> why
>>>> 1.01 is misbehaving on windows, if Erlang is problematic  and to use as
>>>> future tests for new releases of Couchdb on windows.
>>>>
>>>> I downloaded the latest full Erlang windows binary as my base case
>>>> (currently R14B) and wrote the following very simple bit of code as my
>> first
>>>> test
>>>>
>>>> <code>
>>>> -module(filetest).
>>>> -export([main/0]).
>>>> main() ->
>>>> {ok, Binary}=file:read_file("blanks"),
>>>> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
>>>>
>>>> loop(1000, WriteDescr,Binary),
>>>> file:close(WriteDescr).
>>>>
>>>> loop(0,_NotNeeded,_NotNeeded) -> ok;
>>>>
>>>> loop(N,WriteDescr,Binary) ->
>>>> file:write(WriteDescr,Binary),
>>>> loop(N-1,WriteDescr,Binary).
>>>> </code>
>>>>
>>>> Note blanks is a zero filled file of about 2.2MB that i originally used
>> to
>>>> confirm that i had the same issues as Peter first reported.
>>>>
>>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>>> delete filetest.txt
>>>> Run again with a loop of 2000 and you get a file in excess of 4GB
>> ..........
>>>> nice .......... but slow
>>>> delete filetest.txt
>>>> Run again with a loop of 4000 and you get a file in excess of 8GB
>>>> ...........nice............but very slow
>>>>
>>>> Now everything seems fine and dandy but .......
>>>>
>>>> Delete filetest.txt
>>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>>> Run again with a loop of 2000 and you get a file in excess of 4GB
>> ..........
>>>> nice .......... but slow
>>>> Run again with a loop of 2000 and the program begins to "append" from
the
>>>> beginning of the file so the file does not grow but instead gets
>>>> overwritten. This behaviour seems to occur at the 4GB file size on
change
>> of
>>>> file descriptor ............. not nice ....... not good.
>>>>
>>>> I've had a look at the way Erlang is compiled for windows and there are
a
>>>> couple of flags pertaining to large file support. Over the next few
days
>> I
>>>> will attempt my own compilation of Erlang with these flags set and
report
>>>> back. (Need to get hold of VC++ first)
>>>>
>>>> regards
>>>>
>>>> Cliff
>>>>
>>>>
>>>
>>> Excellent work!!
>>> And 0.11 works because it doesn't open files in append mode.
>>>
>>> The file driver uses SetFilePointer in append mode to write to eof.
>>>
>>> DWORD WINAPI SetFilePointer(
>>> __in HANDLE hFile,
>>> __in LONG lDistanceToMove,
>>> __inout_opt PLONG lpDistanceToMoveHigh,
>>> __in DWORD dwMoveMethod
>>> );
>>>
>>> MS documentation states that "To work with file pointers that are
>>> larger than a single LONG value, it is easier to use the
>>> SetFilePointerEx function."
>>> (http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
>>> and the driver passes NULL to lpDistanceToMoveHigh
>>> Suspicious.
>>>
>>> I'll digg a little here.
>>>
>>> -juhani
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message