incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Chitipothu <anandol...@gmail.com>
Subject How to speedup view generation?
Date Sat, 30 Oct 2010 13:52:30 GMT
Hi,

I'm trying to setup a couchdb database with 14M documents. The view
generation is taking too long. It is running at the rate of 22
docs/sec right now. At this rate it will take 7days to build the view,
which is too slow and I expect the speed to go down further as the
view file size increase.

Is there any way to speedup the view generation? Is it possible to
parallelize it?

Here is what I've done till now:

* I tried writing the view in javascript, python and erlang and the
python version seems to be working faster, so I'm using that.
* I've presorted the documents by key before loading, so the view
updater must be reading the docs from disk sequentially.
* I've put the db file and view files on separate disks.

You can look at my view and sample docs here:
http://github.com/anandology/ol-couch/tree/master/python/views/lists
http://github.com/anandology/ol-couch/blob/master/sample_docs.txt

I'm working with couchdb 1.0.1 on Linux server.

I'm struggling to improve this from past 2 weeks with no luck. Do you
have any tips to help me?

Anand

Mime
View raw message