jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jukka Zitting <jukka.zitt...@gmail.com>
Subject Re: Asynchronous indexing consistency
Date Wed, 29 May 2013 11:36:34 GMT

On Wed, May 29, 2013 at 12:28 PM, Thomas Mueller <mueller@adobe.com> wrote:
> What would happen if you stop a cluster node for a long time (for example
> 1 day)? Would async indexing be done on another cluster node? If yes, I
> guess we need a way to ensure that's the case. If not, then the problem
> might be that old revisions are no longer available.

There could be various reasons for why an indexer might not be
available for an extended amount of time, so I think in any case we
need some mechanism for it to pick up from where it left. As you
mentioned, journaled observation will need some similar mechanism.

I see at least the following options:

a) If, like in the Segment and H2 MKs, we could rely on the MKs
supporting cheap copies and diffs across subtrees, we could implement
this without API changes by keeping a copy of the last indexed/seen
state of the repository in a hidden subtree. The indexer would refresh
this copy on each index update, and could thus always know what
content has already been indexed. Unfortunately there probably isn't
any easy way to do this in the MongoMK.

b) Have some way to mark specific revisions as ones that should be
kept around for a longer time (e.g. using a lease mechanism). The
indexer could then store such a revision id as a part of an index
update as a record of what content was last indexed.

c) Keep a log of all changes since the last index update. This is
probably the least attractive solution as it adds quite a bit of write
overhead and, unless the log is maintained by the MK, we'd still have
to worry about potential lost updates.


Jukka Zitting

View raw message