Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
Sender: calzakk@gmail.com
In-Reply-To: <C520D95E-DB72-4EE4-939C-2AEC02A6C8B8@apache.org>
References: <CAL5U6OGyDi+LibVGLCVBJBSS_6LuE1P=GyWTgT3yCebk6rE7nA@mail.gmail.com>
 <C520D95E-DB72-4EE4-939C-2AEC02A6C8B8@apache.org>
From: Graham Bull <calzakk+couchdb@gmail.com>
Date: Tue, 1 Nov 2016 21:39:49 +0000
Message-ID: <CAL5U6OGA0S7NeGAL79+H1bOMYpRuZtEsqh5CA_5NhK0qMNb1kA@mail.gmail.com>
Subject: Re: Indexes are inaccessible while indexers are running
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=94eb2c05e916f907250540442901
archived-at: Tue, 01 Nov 2016 21:39:56 -0000

--94eb2c05e916f907250540442901
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Adam,

Thanks for your reply. I'll have a look at the option when I get a chance,
looks like it could be useful.

I'm a little concerned though. You say the indexes don't update
automatically. What happens when values in indexed fields change?
Presumably the indexes aren't therefore updated, in which case they must be
periodically manually recreated?

I'm running the latest version, 2.0.0, on a decent desktop machine (i7, 8
cores, 16GB RAM, SSDs). Going forward, our application would run on higher
spec servers.

FYI, the indexers actually took close to 4 hours to do all the indexing,
not the 2.5 I thought it would take. One thing I didn't mention was that
I'd created 4 separate indexes, and it's possible we'd need more. It's
looking likely that CouchDB isn't a good fit for what we're doing, which is
a shame because it has lots of positives.

Graham


On 1 November 2016 at 16:08, Adam Kocoloski <kocolosk@apache.org> wrote:

> Hi Graham, the indexes don=E2=80=99t update automatically, but it is poss=
ible to
> prime the indexers by issuing a query to the view at any point during the
> import. One interesting option for you is the "?stale=3Dupdate_after" fla=
g,
> which will respond with the current state of the view index and trigger a
> background update of the indexes after the fact:
>
> GET /<db>/_design/<ddoc>/_view/<view>?stale=3Dupdate_after
>
> You could also add a &limit=3D0 if you=E2=80=99re only interested in prim=
ing the
> indexers.
>
> As far as indexing performance is concerned =E2=80=A6 ~4500 docs/second i=
sn=E2=80=99t
> awesome, but the devil is in the details: how many times is each document
> indexed? Does the server have adequate CPU and IO? Are you running 2.0 or
> one of the 1.x versions?
>
> I can dig up some benchmarks but I=E2=80=99m certain I=E2=80=99ve seen (L=
inux) systems
> index several times faster than that. I haven=E2=80=99t seen a lot of ext=
ensive
> performance testing on Windows though. Cheers,
>
> Adam
>
> > On Oct 31, 2016, at 7:41 AM, Graham Bull <calzakk+couchdb@gmail.com>
> wrote:
> >
> > Hello,
> >
> > I'm currently evaluating CouchDB (and other NoSQL databases).
> >
> > I have a number of databases of various sizes. After restarting the
> CouchDB
> > service (I'm on Windows) eight "indexer" tasks started running on the
> > largest database (40 million documents), which was recently imported.
> >
> > After 30 minutes the progress on all tasks is 20%. In the meantime I
> can't
> > run any queries using the database's indexes. At this rate, it'll take
> > around 2.5 hours to index the entire database.
> >
> > Presumably, when indexes are created, they're initially empty? And the
> > indexer tasks are required to do the actual indexing? If so, then the
> > performance is pretty bad. It took nearly 2 hours to import the 40
> million
> > records. Add on index creation, and you're looking at 4.5 hours. Withou=
t
> > mentioning other relational and NoSQL databases by name, or giving any
> > stats, CouchDB's import and indexing performance is pretty bad in
> > comparison.
> >
> > Is there a way to force the indexers to run immediately after importing
> the
> > data, and to query the indexing status so that my app can wait until it=
's
> > completed?
> >
> > Thanks in advance,
> >
> > Graham
>
>

--94eb2c05e916f907250540442901--