couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joan Touzet <woh...@apache.org>
Subject Re: Indexes are inaccessible while indexers are running
Date Tue, 01 Nov 2016 20:31:24 GMT
Hey Adam,

Are there any plans to open source any of the benchmarking code
Cloudant has done in the past for performance testing? If so I could
give the new Windows build a benchmark against Linux on the same
hardware.

If not I could try throwing something together in tsung, but it'd be
pretty low priority work for me.

-Joan

----- Original Message -----
> From: "Adam Kocoloski" <kocolosk@apache.org>
> To: user@couchdb.apache.org
> Sent: Tuesday, November 1, 2016 12:08:13 PM
> Subject: Re: Indexes are inaccessible while indexers are running
> 
> Hi Graham, the indexes don’t update automatically, but it is possible
> to prime the indexers by issuing a query to the view at any point
> during the import. One interesting option for you is the
> "?stale=update_after" flag, which will respond with the current
> state of the view index and trigger a background update of the
> indexes after the fact:
> 
> GET /<db>/_design/<ddoc>/_view/<view>?stale=update_after
> 
> You could also add a &limit=0 if you’re only interested in priming
> the indexers.
> 
> As far as indexing performance is concerned … ~4500 docs/second isn’t
> awesome, but the devil is in the details: how many times is each
> document indexed? Does the server have adequate CPU and IO? Are you
> running 2.0 or one of the 1.x versions?
> 
> I can dig up some benchmarks but I’m certain I’ve seen (Linux)
> systems index several times faster than that. I haven’t seen a lot
> of extensive performance testing on Windows though. Cheers,
> 
> Adam
> 
> > On Oct 31, 2016, at 7:41 AM, Graham Bull
> > <calzakk+couchdb@gmail.com> wrote:
> > 
> > Hello,
> > 
> > I'm currently evaluating CouchDB (and other NoSQL databases).
> > 
> > I have a number of databases of various sizes. After restarting the
> > CouchDB
> > service (I'm on Windows) eight "indexer" tasks started running on
> > the
> > largest database (40 million documents), which was recently
> > imported.
> > 
> > After 30 minutes the progress on all tasks is 20%. In the meantime
> > I can't
> > run any queries using the database's indexes. At this rate, it'll
> > take
> > around 2.5 hours to index the entire database.
> > 
> > Presumably, when indexes are created, they're initially empty? And
> > the
> > indexer tasks are required to do the actual indexing? If so, then
> > the
> > performance is pretty bad. It took nearly 2 hours to import the 40
> > million
> > records. Add on index creation, and you're looking at 4.5 hours.
> > Without
> > mentioning other relational and NoSQL databases by name, or giving
> > any
> > stats, CouchDB's import and indexing performance is pretty bad in
> > comparison.
> > 
> > Is there a way to force the indexers to run immediately after
> > importing the
> > data, and to query the indexing status so that my app can wait
> > until it's
> > completed?
> > 
> > Thanks in advance,
> > 
> > Graham
> 
> 

Mime
View raw message