incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <d...@muse.net.nz>
Subject Re: Windows Server 2003 4GB file issue not solved
Date Wed, 16 Feb 2011 09:00:22 GMT
On 16 February 2011 01:12, Rob Pettefar <rpettefar@gpslsolutions.com> wrote:
> Hi guys
> I am using Windows 7 Ultimate and I managed to get round the 4GB database
> limit using this installer:
> https://github.com/dch/couchdb/downloads/setup-couchdb-1.0.2_otp_R14B01_spidermonkey_1.8.5.exe
> With this I have a 5.8GB database that is happily running, even if I restart
> the service.
>
> However when this was applied to a Windows Server 2003 Standard Edition
> machine the 4GB limit still seems to exist.

Hi Rob

If this is still causing errors then it's pretty serious -
over-written (lost) data at the head of the couch db file is the worst
case scenario. The 1.0.2 binary was checked for this issue on both 32
and 64bit platforms & came up clean.

Firstly more detail please -  William of Ockham[3] is betting on the first two
- was this a clean install of CouchDB into a new folder, or
overwriting an existing install (not recommended)?
- are you running on an NTFS volume for /var/lib/couchdb/*?
- what caused you to see a 4GiB limit? errors, logs, unexpected behaviour?

History -
The 4GiB error is caused by inability in erlang 13B04 and earlier
versions to append data to a file that is already > 4 GiB on windows
only. This means that:

1) A running CouchDB instance can't grow a db file from < 4 GiB to
over 4 GiB without crashing.

2) On restart, the first time that  > 4GiB db file has a reason to
write - e.g. you add a doc - it starts over-writing not from the end
(append) but from the beginning of the DB = lost header = possible
lost data (I think depending if compaction is needed or not).

This was first resolved in 14B01 only with Juhani's patch.

3 checks you can do on a *clean* folder install of CouchDB

1) confirm what erl/werl version you are using
2) confirm there are no issues at erlang level using werl.exe shell
- create a large file (I use one around 128MiB) & use Juhani's quick
checker [1] below from the erlang shell to drive it past 4GiB. This
should be successful. Note the size & the content of the header of the
file.
- re-run the quick checker again - each time a segment is written you
would normally (e.g. in 1 above) see the file size grow. It does not,
and examining the header shows you have overwritten it.

3) less direct but more couchy:
- make a large DB just under 4GiB = 4096 * 1024 * 1024 -1 max size. I
upload many docs with large attachments for this.
- stop couchdb & record header and filesize of db.
- upload more docs & increase the db size past 4GiB -- if you can....
- watch couch/erlang die if error is present
http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
- stop couchdb & record header and filesize of db.
- start couchdb & relaunch the script again.
- confirm header changed, and filesize has not, even though docs were written.

> Is there something funky with Server 2003 that would cause this?

Not to my knowledge. It would be an OS bug in this case if the API is
inconsistent between XP/2003/2008 32 and 64 bit.

> Is there a binary or installer available that would work instead?
At the moment I suspect

> Also do database files that grow to the magical 4GB limit become corrupt or
> are they usable on other setups without this issue?

All fine so long as you don't restart. Best is to stop couch, move the
db files safely, and upgrade.

> (I have to ask as the 2003 Server machine is over in china and will take an
> age to transfer it for examination)
>
> Any help on this matter would be greatly appreciated.
> Rob
>

Look for me on irc d_ch & we can go through the gory bits. I'm in UTC
+ 12 but very keen to help here.
A+
Dave

[3] http://www.phys.ncku.edu.tw/mirrors/physicsfaq/General/occam.html
[2] http://friendpaste.com/6moY1vyUyIsIX4t5N4hZ1B?rev=626638353230
[1] filetest.erl

-module(filetest).
-export([main/0]).
main() ->
{ok, Binary}=file:read_file("128MiB"),
{ok, WriteDescr} = file:open("grow_past_4_GiB.file", [raw, append]),

loop(1000, WriteDescr,Binary),
file:close(WriteDescr).

loop(0,_NotNeeded,_NotNeeded) -> ok;

loop(N,WriteDescr,Binary) ->
file:write(WriteDescr,Binary),
io:format("wrote ~w\n", [N]),
loop(N-1,WriteDescr,Binary).

Mime
View raw message