incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <randall.le...@gmail.com>
Subject Re: Volume Test - 2 million documents
Date Wed, 13 Oct 2010 00:39:20 GMT
Comments inline.

On Tue, Oct 12, 2010 at 17:16, Tracy Flynn <couchdb@thisonejustforme.com> wrote:
>
> Firstly any thoughts or hints on my larger benchmark (Part 2) ?
>
> Is it naive to hope to speed up the first creation of the view by using map functions
of the form 'emit([key],null)' and then using 'include_docs' on queries?
>

No. This sounds very reasonable and accurate. You didn't specify
whether or not your laptop has an SSD, but regardless I expect you
will probably be disk bound. Therefore, reducing the disk writes by
not duplicating document bodies in the view index will speed up
indexing.

>
> Is there any way to control the checkpointing of views when creating the view for the
first time - I'm guessing I'm looking at many hours to create a single view on 2 million documents.
>

Probably. Thankfully this will only happen once.
Maybe experiment with doing batches of inserts followed by a limit=0
view query to refresh the index. While you do have a lot of RAM this
might improve your fs cache hit rate by ensuring documents passed to
the view index are recently written and still hot. I could be totally
wrong, though.

Good luck!
Randall

Mime
View raw message