couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cliff Williams <cliffywi...@aol.com>
Subject Re: Couchdb crashes on Windows
Date Thu, 30 Sep 2010 10:01:11 GMT
  Juhani,

That was very very quick. Well done to you !!!!!!!!!

@Dave this is what I am using
http://www.erlang.org/doc/installation_guide/users_guide.html

the flags that I was interested in (but now superseded by Juhani's patch 
& Randall's efforts) were
LFS_CFLAGS - Large file support C compiler flags.
LFS_LDFLAGS - Large file support linker flags.
LFS_LIBS - Large file support libraries.

very best regards

Cliff



On 30/09/10 09:29, Juhani Ränkimies wrote:
> You remember correctly. efile_seek uses SetFilePointerEx variant, but
> efile_write and efile_writev call SetFilePointer when in append mode .
>
> I searched a little and found that it's possible to have WriteFile
> write at EOF without first setting the file pointer (one less system
> call). (http://msdn.microsoft.com/en-us/library/aa365747%28v=VS.85%29.aspx,
> at the bottom of the page)
>
> Did a quick test  and it seems to work.
> http://github.com/juranki/otp/commit/9397777751de1ef20de29a911162da2385801c52
>
> -juhani
>
> On Thu, Sep 30, 2010 at 9:37 AM, Randall Leeds<randall.leeds@gmail.com>  wrote:
>> Oh, no! I suspected the change to append to be the issue looking into this
>> yesterday, too. I looked in the windows file driver (R13B05) and found the
>> seek function (efile_seek or something??), but thought I remembered it
>> properly taking 64bit values and using SetFilePointerEx. I also recall
>> finding the append path calling the this function, but maybe I made that
>> part up. I didn't get a chance to fully stamp this out either, but I think
>> we're on the right track.
>>> On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams<cliffywills@aol.com>
>> wrote:
>>>>   Dave,
>>>>
>>>> I hope you are well
>>>>
>>>> I too began to look at this a couple of nights ago.
>>>>
>>>> The process that I was going to follow;
>>>>     Establish a base case
>>>>     Write some file IO tests
>>>>     Apply these tests to the Couchdb environment and see what differences
>>>> popped out.
>>>>
>>>> At the very least I thought that I could produce some tests to ascertain
>> why
>>>> 1.01 is misbehaving on windows, if Erlang is problematic  and to use as
>>>> future tests for new releases of Couchdb on windows.
>>>>
>>>> I downloaded the latest full Erlang windows binary as my base case
>>>> (currently R14B) and wrote the following very simple bit of code as my
>> first
>>>> test
>>>>
>>>> <code>
>>>> -module(filetest).
>>>> -export([main/0]).
>>>> main() ->
>>>> {ok, Binary}=file:read_file("blanks"),
>>>> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
>>>>
>>>> loop(1000, WriteDescr,Binary),
>>>> file:close(WriteDescr).
>>>>
>>>> loop(0,_NotNeeded,_NotNeeded) ->  ok;
>>>>
>>>> loop(N,WriteDescr,Binary) ->
>>>> file:write(WriteDescr,Binary),
>>>> loop(N-1,WriteDescr,Binary).
>>>> </code>
>>>>
>>>> Note blanks is a zero filled file of about 2.2MB that i originally used
>> to
>>>> confirm that i had the same issues as Peter first reported.
>>>>
>>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>>> delete filetest.txt
>>>> Run again with a loop of 2000 and you get a file in excess of 4GB
>> ..........
>>>> nice .......... but slow
>>>> delete filetest.txt
>>>> Run again with a loop of 4000 and you get a file in excess of 8GB
>>>> ...........nice............but very slow
>>>>
>>>> Now everything seems fine and dandy but .......
>>>>
>>>> Delete filetest.txt
>>>> Run with a loop of 10 and you get a twenty MB file ........... nice
>>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
>>>> Run again with a loop of 2000 and you get a file in excess of 4GB
>> ..........
>>>> nice .......... but slow
>>>> Run again with a loop of 2000 and the program begins to "append" from the
>>>> beginning of the file so the file does not grow but instead gets
>>>> overwritten. This behaviour seems to occur at the 4GB file size on change
>> of
>>>> file descriptor ............. not nice ....... not good.
>>>>
>>>> I've had a look at the way Erlang is compiled for windows and there are a
>>>> couple of flags pertaining to large file support. Over the next few days
>> I
>>>> will attempt my own compilation of Erlang with these flags set and report
>>>> back. (Need to get hold of VC++ first)
>>>>
>>>> regards
>>>>
>>>> Cliff
>>>>
>>>>
>>> Excellent work!!
>>> And 0.11 works because it doesn't open files in append mode.
>>>
>>> The file driver uses SetFilePointer in append mode to write to eof.
>>>
>>> DWORD WINAPI SetFilePointer(
>>> __in HANDLE hFile,
>>> __in LONG lDistanceToMove,
>>> __inout_opt PLONG lpDistanceToMoveHigh,
>>> __in DWORD dwMoveMethod
>>> );
>>>
>>> MS documentation states that "To work with file pointers that are
>>> larger than a single LONG value, it is easier to use the
>>> SetFilePointerEx function."
>>> (http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
>>> and the driver passes NULL to lpDistanceToMoveHigh
>>> Suspicious.
>>>
>>> I'll digg a little here.
>>>
>>> -juhani

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message