Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of cliffywills@aol.com
 designates 205.188.105.147 as permitted sender)
Message-ID: <4CA3520C.7050606@aol.com>
Date: Wed, 29 Sep 2010 15:49:48 +0100
From: Cliff Williams <cliffywills@aol.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.11pre) Gecko/20100922 Shredder/3.1.5pre
MIME-Version: 1.0
To: user@couchdb.apache.org
Subject: Re: Couchdb crashes on Windows
References: <AANLkTi=fy3phwQo1+bis9E=z+1HJR5wTBTK2-skOQ1Z8@mail.gmail.com>
	<A9DC438D6CAFFF4BBB29958144440881012018E26B0F@VS-EX-01.intra.vpro.nl>
	<AANLkTinxDxJJAN7qh86Aw2P=vpeOzYB=u4bJgQmg-sHR@mail.gmail.com>
	<A9DC438D6CAFFF4BBB29958144440881012018E26B10@VS-EX-01.intra.vpro.nl>
	<AANLkTi=d2-Jv5m3O_jHkZFRkstvxBBO3oHstaxmE1-jS@mail.gmail.com>
	<4C9DDCE2.3050307@aol.com>
	<AANLkTikncP8X3J81UVYS9=i1m24sGz=GcgY8RrXkjVsi@mail.gmail.com>
	<4C9DE9D0.5090002@aol.com>
	<AANLkTimPQHSEEBPHwiJ+HWxaKd=WoiEFKWpyvHsR5xPu@mail.gmail.com>
	<AANLkTikTVBObEqKHkMVBMDpuWVBNA+CAf6iXPRAL0wjs@mail.gmail.com>
	<AANLkTikcs=uywFmtTLtTDGnwNOie_=MQFd9d7VHU2yUf@mail.gmail.com>
	<AANLkTimymf1DCd7vRh0YS1f=StGw9E4OmceRn++Bqku=@mail.gmail.com>
 <AANLkTims_JQYhZYhpaWLumbvAjLN5EhAGk6es_PthCDG@mail.gmail.com>
In-Reply-To: <AANLkTims_JQYhZYhpaWLumbvAjLN5EhAGk6es_PthCDG@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------010606010603030404010407"
X-AOL-SCOLL-URL_COUNT: 0  
x-aol-sid: 3039ac1d33c24ca3520e76f6
X-AOL-IP: 92.1.146.206

--------------010606010603030404010407
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

  Dave,

I hope you are well

I too began to look at this a couple of nights ago.

The process that I was going to follow;
     Establish a base case
     Write some file IO tests
     Apply these tests to the Couchdb environment and see what 
differences popped out.

At the very least I thought that I could produce some tests to ascertain 
why 1.01 is misbehaving on windows, if Erlang is problematic  and to use 
as future tests for new releases of Couchdb on windows.

I downloaded the latest full Erlang windows binary as my base case 
(currently R14B) and wrote the following very simple bit of code as my 
first test

<code>
-module(filetest).
-export([main/0]).
main() ->
{ok, Binary}=file:read_file("blanks"),
{ok, WriteDescr} = file:open(filetest.txt, [raw, append]),

loop(1000, WriteDescr,Binary),
file:close(WriteDescr).

loop(0,_NotNeeded,_NotNeeded) -> ok;

loop(N,WriteDescr,Binary) ->
file:write(WriteDescr,Binary),
loop(N-1,WriteDescr,Binary).
</code>

Note blanks is a zero filled file of about 2.2MB that i originally used 
to confirm that i had the same issues as Peter first reported.

Run with a loop of 10 and you get a twenty MB file ........... nice
Run again with a loop of 10 and you get a Forty MB file ......... nice
delete filetest.txt
Run again with a loop of 2000 and you get a file in excess of 4GB 
.......... nice .......... but slow
delete filetest.txt
Run again with a loop of 4000 and you get a file in excess of 8GB 
...........nice............but very slow

Now everything seems fine and dandy but .......

Delete filetest.txt
Run with a loop of 10 and you get a twenty MB file ........... nice
Run again with a loop of 10 and you get a Forty MB file ......... nice
Run again with a loop of 2000 and you get a file in excess of 4GB 
.......... nice .......... but slow
Run again with a loop of 2000 and the program begins to "append" from 
the beginning of the file so the file does not grow but instead gets 
overwritten. This behaviour seems to occur at the 4GB file size on 
change of file descriptor ............. not nice ....... not good.

I've had a look at the way Erlang is compiled for windows and there are 
a couple of flags pertaining to large file support. Over the next few 
days I will attempt my own compilation of Erlang with these flags set 
and report back. (Need to get hold of VC++ first)

regards

Cliff


On 29/09/10 01:02, Dave Cottlehuber wrote:
> On 26 September 2010 10:11, Peter Somers<petsomers@gmail.com>  wrote:
>> Hi Dave,
>>
>> No luck. Couchdb crashed again :-(
>>
>> I really hope someone can help me on this issue...
>>
>> Best regards,
>>
>> Peter
> Hi Peter
>
> There may be 2 things going on here.
>
> - your current crash appears to be due to file corruption;
>   - {{badmatch,{error,file_corruption}},
> this is after couch reads an md5 hash in read_raw_iolist_int - maybe
> due to going over length, I can't tell.
>
> - your first crash is the one I can replicate - the whole VM dies with
> a memory allocation issue. I can replicate this on my win7 enterprise
> couch running 1.01, but not 0.11.0.
>
> I can load the DB up to this size; 0xffd9a0f6d, couchdb dies:
>
>      {{badmatch,{error,enomem}}, [{couch_file,read_raw_iolist_int,3},
> {couch_file,handle_call,3},
>
> Restarting windows/couchdb&  then curl http://localhost:5984/fatso yields
> {"error":"kill","reason":"{gen_server,call,
>      [couch_server,
>       {open,<<\"fatso\">>,
>           [{user_ctx,
>                {user_ctx,null,
>                    [<<\"_admin\">>],
>                    <<\"{couch_httpd_auth,
> default_authentication_handler}\">>}}]},
>       infinity]}"}
>
> Anyway some more work ahead to identify exactly where this comes from
> - thanks for raising the jira.
>
> cheers
> Dave
>

--------------010606010603030404010407--