manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Scaling in MCF
Date Thu, 13 Nov 2014 15:13:17 GMT
Hi Aeham,

Frequency of reindexing and analysis is basically settable by parameter on
each individual table.  See the how-to-build-and-deploy page for details.
The details of how many "events" have occurred since the cluster was
started (or actually, since zookeeper was started) are tracked in a
cross-cluster manner.  If you are using zookeeper, and you *don't* clean up
zookeeper's data, the tracking should persist even across shutdowns and
restarts, since it's kept via a persistent node.

If this is not what you are seeing, we should try to figure out why not.
Perhaps you just need to set the reindexing parameter for the jobqueue
table to a higher number, or maybe something isn't working as designed.

Karl


On Thu, Nov 13, 2014 at 9:44 AM, Aeham Abushwashi <
aeham.abushwashi@exonar.com> wrote:

> Looking at the 9.2 postgresql docs on REINDEX (
> http://www.postgresql.org/docs/9.2/static/sql-reindex.html),I think you're
> right in that it's not strictly necessary as an ongoing operation but that
> it may help performance a little by reducing bloat and grouping pages
> together (http://www.postgresql.org/docs/9.2/static/routine-reindex.html).
> Perhaps it's useful for Manifold to have the ability to invoke REINDEX but
> that it should just do it less often?
>
> I'll look into this a bit more and also keep a closer watch on REINDEXing
> activity in the upcoming tests...
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message