couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Somers <petsom...@gmail.com>
Subject Re: Couchdb crashes on Windows
Date Fri, 01 Oct 2010 15:14:34 GMT
Very nice work indeed!

However, as I am no erlang programmer and we have a project that is going
live soon, what are the chances that this fix will result in a new windows
build that can be downloaded and installed soon?

I don't want to push anyone, I just need some realistic time frame.
What is the conventional process in this case?


Thank you all.

Peter


2010/9/30 Randall Leeds <randall.leeds@gmail.com>

> Niiice work!
> > You remember correctly. efile_seek uses SetFilePointerEx variant, but
> > efile_write and efile_writev call SetFilePointer when in append mode .
> >
> > I searched a little and found that it's possible to have WriteFile
> > write at EOF without first setting the file pointer (one less system
> > call). (
> http://msdn.microsoft.com/en-us/library/aa365747%28v=VS.85%29.aspx
> ,
> > at the bottom of the page)
> >
> > Did a quick test and it seems to work.
> >
>
> http://github.com/juranki/otp/commit/9397777751de1ef20de29a911162da2385801c52
> >
> > -juhani
> >
> > On Thu, Sep 30, 2010 at 9:37 AM, Randall Leeds <randall.leeds@gmail.com>
> wrote:
> >> Oh, no! I suspected the change to append to be the issue looking into
> this
> >> yesterday, too. I looked in the windows file driver (R13B05) and found
> the
> >> seek function (efile_seek or something??), but thought I remembered it
> >> properly taking 64bit values and using SetFilePointerEx. I also recall
> >> finding the append path calling the this function, but maybe I made that
> >> part up. I didn't get a chance to fully stamp this out either, but I
> think
> >> we're on the right track.
> >>> On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams <cliffywills@aol.com>
> >> wrote:
> >>>>  Dave,
> >>>>
> >>>> I hope you are well
> >>>>
> >>>> I too began to look at this a couple of nights ago.
> >>>>
> >>>> The process that I was going to follow;
> >>>>    Establish a base case
> >>>>    Write some file IO tests
> >>>>    Apply these tests to the Couchdb environment and see what
> differences
> >>>> popped out.
> >>>>
> >>>> At the very least I thought that I could produce some tests to
> ascertain
> >> why
> >>>> 1.01 is misbehaving on windows, if Erlang is problematic  and to use
> as
> >>>> future tests for new releases of Couchdb on windows.
> >>>>
> >>>> I downloaded the latest full Erlang windows binary as my base case
> >>>> (currently R14B) and wrote the following very simple bit of code as
my
> >> first
> >>>> test
> >>>>
> >>>> <code>
> >>>> -module(filetest).
> >>>> -export([main/0]).
> >>>> main() ->
> >>>> {ok, Binary}=file:read_file("blanks"),
> >>>> {ok, WriteDescr} = file:open(filetest.txt, [raw, append]),
> >>>>
> >>>> loop(1000, WriteDescr,Binary),
> >>>> file:close(WriteDescr).
> >>>>
> >>>> loop(0,_NotNeeded,_NotNeeded) -> ok;
> >>>>
> >>>> loop(N,WriteDescr,Binary) ->
> >>>> file:write(WriteDescr,Binary),
> >>>> loop(N-1,WriteDescr,Binary).
> >>>> </code>
> >>>>
> >>>> Note blanks is a zero filled file of about 2.2MB that i originally
> used
> >> to
> >>>> confirm that i had the same issues as Peter first reported.
> >>>>
> >>>> Run with a loop of 10 and you get a twenty MB file ........... nice
> >>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
> >>>> delete filetest.txt
> >>>> Run again with a loop of 2000 and you get a file in excess of 4GB
> >> ..........
> >>>> nice .......... but slow
> >>>> delete filetest.txt
> >>>> Run again with a loop of 4000 and you get a file in excess of 8GB
> >>>> ...........nice............but very slow
> >>>>
> >>>> Now everything seems fine and dandy but .......
> >>>>
> >>>> Delete filetest.txt
> >>>> Run with a loop of 10 and you get a twenty MB file ........... nice
> >>>> Run again with a loop of 10 and you get a Forty MB file ......... nice
> >>>> Run again with a loop of 2000 and you get a file in excess of 4GB
> >> ..........
> >>>> nice .......... but slow
> >>>> Run again with a loop of 2000 and the program begins to "append" from
> the
> >>>> beginning of the file so the file does not grow but instead gets
> >>>> overwritten. This behaviour seems to occur at the 4GB file size on
> change
> >> of
> >>>> file descriptor ............. not nice ....... not good.
> >>>>
> >>>> I've had a look at the way Erlang is compiled for windows and there
> are
> a
> >>>> couple of flags pertaining to large file support. Over the next few
> days
> >> I
> >>>> will attempt my own compilation of Erlang with these flags set and
> report
> >>>> back. (Need to get hold of VC++ first)
> >>>>
> >>>> regards
> >>>>
> >>>> Cliff
> >>>>
> >>>>
> >>>
> >>> Excellent work!!
> >>> And 0.11 works because it doesn't open files in append mode.
> >>>
> >>> The file driver uses SetFilePointer in append mode to write to eof.
> >>>
> >>> DWORD WINAPI SetFilePointer(
> >>> __in HANDLE hFile,
> >>> __in LONG lDistanceToMove,
> >>> __inout_opt PLONG lpDistanceToMoveHigh,
> >>> __in DWORD dwMoveMethod
> >>> );
> >>>
> >>> MS documentation states that "To work with file pointers that are
> >>> larger than a single LONG value, it is easier to use the
> >>> SetFilePointerEx function."
> >>> (http://msdn.microsoft.com/en-us/library/aa365541%28v=vs.85%29.aspx)
> >>> and the driver passes NULL to lpDistanceToMoveHigh
> >>> Suspicious.
> >>>
> >>> I'll digg a little here.
> >>>
> >>> -juhani
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message