From user-return-13001-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Sep 30 05:18:48 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 66225 invoked from network); 30 Sep 2010 05:18:47 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 30 Sep 2010 05:18:47 -0000 Received: (qmail 62075 invoked by uid 500); 30 Sep 2010 05:18:46 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 61695 invoked by uid 500); 30 Sep 2010 05:18:43 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 61687 invoked by uid 99); 30 Sep 2010 05:18:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 05:18:42 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.83.52] (HELO mail-gw0-f52.google.com) (74.125.83.52) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 05:18:34 +0000 Received: by gwj20 with SMTP id 20so741810gwj.11 for ; Wed, 29 Sep 2010 22:18:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.211.7 with SMTP id gm7mr2144874qcb.72.1285823892494; Wed, 29 Sep 2010 22:18:12 -0700 (PDT) Received: by 10.229.239.66 with HTTP; Wed, 29 Sep 2010 22:18:12 -0700 (PDT) In-Reply-To: <4CA3520C.7050606@aol.com> References: <4C9DDCE2.3050307@aol.com> <4C9DE9D0.5090002@aol.com> <4CA3520C.7050606@aol.com> Date: Thu, 30 Sep 2010 08:18:12 +0300 Message-ID: Subject: Re: Couchdb crashes on Windows From: =?ISO-8859-1?Q?Juhani_R=E4nkimies?= To: user@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Wed, Sep 29, 2010 at 5:49 PM, Cliff Williams wrote= : > =A0Dave, > > I hope you are well > > I too began to look at this a couple of nights ago. > > The process that I was going to follow; > =A0 =A0Establish a base case > =A0 =A0Write some file IO tests > =A0 =A0Apply these tests to the Couchdb environment and see what differen= ces > popped out. > > At the very least I thought that I could produce some tests to ascertain = why > 1.01 is misbehaving on windows, if Erlang is problematic =A0and to use as > future tests for new releases of Couchdb on windows. > > I downloaded the latest full Erlang windows binary as my base case > (currently R14B) and wrote the following very simple bit of code as my fi= rst > test > > > -module(filetest). > -export([main/0]). > main() -> > {ok, Binary}=3Dfile:read_file("blanks"), > {ok, WriteDescr} =3D file:open(filetest.txt, [raw, append]), > > loop(1000, WriteDescr,Binary), > file:close(WriteDescr). > > loop(0,_NotNeeded,_NotNeeded) -> ok; > > loop(N,WriteDescr,Binary) -> > file:write(WriteDescr,Binary), > loop(N-1,WriteDescr,Binary). > > > Note blanks is a zero filled file of about 2.2MB that i originally used t= o > confirm that i had the same issues as Peter first reported. > > Run with a loop of 10 and you get a twenty MB file ........... nice > Run again with a loop of 10 and you get a Forty MB file ......... nice > delete filetest.txt > Run again with a loop of 2000 and you get a file in excess of 4GB .......= ... > nice .......... but slow > delete filetest.txt > Run again with a loop of 4000 and you get a file in excess of 8GB > ...........nice............but very slow > > Now everything seems fine and dandy but ....... > > Delete filetest.txt > Run with a loop of 10 and you get a twenty MB file ........... nice > Run again with a loop of 10 and you get a Forty MB file ......... nice > Run again with a loop of 2000 and you get a file in excess of 4GB .......= ... > nice .......... but slow > Run again with a loop of 2000 and the program begins to "append" from the > beginning of the file so the file does not grow but instead gets > overwritten. This behaviour seems to occur at the 4GB file size on change= of > file descriptor ............. not nice ....... not good. > > I've had a look at the way Erlang is compiled for windows and there are a > couple of flags pertaining to large file support. Over the next few days = I > will attempt my own compilation of Erlang with these flags set and report > back. (Need to get hold of VC++ first) > > regards > > Cliff > > Excellent work!! And 0.11 works because it doesn't open files in append mode. The file driver uses SetFilePointer in append mode to write to eof. DWORD WINAPI SetFilePointer( __in HANDLE hFile, __in LONG lDistanceToMove, __inout_opt PLONG lpDistanceToMoveHigh, __in DWORD dwMoveMethod ); MS documentation states that "To work with file pointers that are larger than a single LONG value, it is easier to use the SetFilePointerEx function." (http://msdn.microsoft.com/en-us/library/aa365541%28v=3Dvs.85%29.aspx) and the driver passes NULL to lpDistanceToMoveHigh Suspicious. I'll digg a little here. -juhani