couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: couchdb memory issues/leaks with validators and 20MB+ json docs
Date Sun, 19 Sep 2010 12:55:14 GMT

On 30 Aug 2010, at 08:34, sgoto wrote:

> Hey everyone,
> 
>     I'm using couchdb to store docs that are somewhat large (20MB+), but
> within the configured max size.
> 
>     Storing the docs isn't a problem, couchdb seems to handle it fine. I am
> having problems when using function validators and couchdb hanging my
> machine after all the memory resources are consumed on PUTs.
> 
>     Below is a quick explanation of the issue I'm seeing.
> 
>     Ideas ?
> 
>     sam
> 
> 
> how to reproduce:
> 
> 1) create a db called testdb
> 
> 2) create an empty javascript validator function
> 
> function(newDoc, oldDoc, user) {}
> 
> 3) create a fake 20MB doc
> 
> if=/dev/zero of=test.mp3 bs=1024 count=20000
> echo "{\"hello\":\"" > test.json; echo `base64 test.mp3` >> test.json;  echo
> "\"}" >> test.json;
> 
> 4) send it to couchdb
> 
> curl -X PUT http://127.0.0.1:5984/testdb/foobar21 -d @test.json
> 
> 5) open a memory/swap monitor and couchdb's binary consume all the memory
> (stopping when the swap memory ends)
> 
> kubuntu's system monitor (memory tab) ||
> top ||
> watch free ||
> 
> 6) remove the javascript validator
> 
> 7) repeat (5) and see how everything is fine
> 
> expected results:
> 
> (5) shouldn't happen. couchdb shouldn't leak memory or consume more memory
> than the size of the doc (20MB).

How much total memory do you have? CouchDB will consume more than the doc size
in memory (I've seen 2-3x) and using a validation function can blow this up more, but
unless you are on a really space constrained VPS, you shouldn't run into swap.

Cheers
Jan
-- 


> 
> -- 
> f u cn rd ths u cn b a gd prgmr !


Mime
View raw message