incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Pettefar <rpette...@gpslsolutions.com>
Subject Re: Windows Server 2003 4GB file issue not solved
Date Wed, 16 Feb 2011 14:13:06 GMT
  Hi there guys
I did a full clean install of everything and things are working nicely 
again.
I think we were getting false negatives when testing with the 4GB 
database file. I didn't know that it would overwrite, I just assumed it 
was append only. This would probably explain why it can't be read properly.

Thanks for your help in resolving this
Rob

On 16/02/2011 09:00, Dave Cottlehuber wrote:
> On 16 February 2011 01:12, Rob Pettefar<rpettefar@gpslsolutions.com>  wrote:
>> Hi guys
>> I am using Windows 7 Ultimate and I managed to get round the 4GB database
>> limit using this installer:
>> https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
>> With this I have a 5.8GB database that is happily running, even if I restart
>> the service.
>>
>> However when this was applied to a Windows Server 2003 Standard Edition
>> machine the 4GB limit still seems to exist.
> Hi Rob
>
> If this is still causing errors then it's pretty serious -
> over-written (lost) data at the head of the couch db file is the worst
> case scenario. The 1.0.2 binary was checked for this issue on both 32
> and 64bit platforms&  came up clean.
>
> Firstly more detail please -  William of Ockham[3] is betting on the first two
> - was this a clean install of CouchDB into a new folder, or
> overwriting an existing install (not recommended)?
> - are you running on an NTFS volume for /var/lib/couchdb/*?
> - what caused you to see a 4GiB limit? errors, logs, unexpected behaviour?
>
> History -
> The 4GiB error is caused by inability in erlang 13B04 and earlier
> versions to append data to a file that is already>  4 GiB on windows
> only. This means that:
>
> 1) A running CouchDB instance can't grow a db file from<  4 GiB to
> over 4 GiB without crashing.
>
> 2) On restart, the first time that>  4GiB db file has a reason to
> write - e.g. you add a doc - it starts over-writing not from the end
> (append) but from the beginning of the DB = lost header = possible
> lost data (I think depending if compaction is needed or not).
>
> This was first resolved in 14B01 only with Juhani's patch.
>
> 3 checks you can do on a *clean* folder install of CouchDB
>
> 1) confirm what erl/werl version you are using
> 2) confirm there are no issues at erlang level using werl.exe shell
> - create a large file (I use one around 128MiB)&  use Juhani's quick
> checker [1] below from the erlang shell to drive it past 4GiB. This
> should be successful. Note the size&  the content of the header of the
> file.
> - re-run the quick checker again - each time a segment is written you
> would normally (e.g. in 1 above) see the file size grow. It does not,
> and examining the header shows you have overwritten it.
>
> 3) less direct but more couchy:
> - make a large DB just under 4GiB = 4096 * 1024 * 1024 -1 max size. I
> upload many docs with large attachments for this.
> - stop couchdb&  record header and filesize of db.
> - upload more docs&  increase the db size past 4GiB -- if you can....
> - watch couch/erlang die if error is present
> http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
> - stop couchdb&  record header and filesize of db.
> - start couchdb&  relaunch the script again.
> - confirm header changed, and filesize has not, even though docs were written.
>
>> Is there something funky with Server 2003 that would cause this?
> Not to my knowledge. It would be an OS bug in this case if the API is
> inconsistent between XP/2003/2008 32 and 64 bit.
>
>> Is there a binary or installer available that would work instead?
> At the moment I suspect
>
>> Also do database files that grow to the magical 4GB limit become corrupt or
>> are they usable on other setups without this issue?
> All fine so long as you don't restart. Best is to stop couch, move the
> db files safely, and upgrade.
>
>> (I have to ask as the 2003 Server machine is over in china and will take an
>> age to transfer it for examination)
>>
>> Any help on this matter would be greatly appreciated.
>> Rob
>>
> Look for me on irc d_ch&  we can go through the gory bits. I'm in UTC
> + 12 but very keen to help here.
> A+
> Dave
>
> [3] http://www.phys.ncku.edu.tw/mirrors/physicsfaq/General/occam.html
> [2] http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
> [1] filetest.erl
>
> -module(filetest).
> -export([main/0]).
> main() ->
> {ok, Binary}=file:read_file("128MiB"),
> {ok, WriteDescr} = file:open("grow_past_4_GiB.file", [raw, append]),
>
> loop(1000, WriteDescr,Binary),
> file:close(WriteDescr).
>
> loop(0,_NotNeeded,_NotNeeded) ->  ok;
>
> loop(N,WriteDescr,Binary) ->
> file:write(WriteDescr,Binary),
> io:format("wrote ~w\n", [N]),
> loop(N-1,WriteDescr,Binary).


Mime
View raw message