jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jukka Zitting <jukka.zitt...@gmail.com>
Subject Re: SegmentNodeStore merge operations
Date Wed, 06 Mar 2013 20:54:18 GMT
Hi,

On Wed, Mar 6, 2013 at 5:33 PM, Thomas Mueller <mueller@adobe.com> wrote:
> I wonder what is the state of the implementation of merge operations for
> segments and journals, and how are merges scheduled?

Basic merging functionality for the in-memory segment store there, see
the o.a.j.oak.plugins.segment.JournalTest for some examples on how
that works. Currently the merge operation needs to be explicitly
triggered.

Before implementing the same mechanism for the MongoDB store, I've
been experimenting with a few alternatives on how to schedule and
process merges. The most straightforward approach is to use periodic
merges with a configurable interval (defaulting to something like one
second), with the option to explicitly trigger extra merges when
needed, and to process merges using our existing rebase logic that
leaves conflict markers in the tree when unresolvable conflicts are
detected.

However, as noted in OAK-633, there are a few conceptual problems with
this approach to processing merges:

a) Since validators and other commit hooks are not run during the
merge, the result can be an internally inconsistent content tree
(dangling references, incorrect permission store, etc.)

b) The presence of conflict markers will prevent further changes to
affected nodes until the conflict gets resolved

c) There's no good way to handle more than one set of conflicts per node

So, apart from problem a (which also affects the new MongoMK), the
current mechanism works fine (i.e. fully parallel writes) as long as
the changes are non-conflicting, but runs into trouble when there are
conflicts.

So far I've come up with the following alternative designs to address
the above problems:

* Use a more aggressive merge algorithm that automatically resolves
all conflicts by throwing away (or storing somewhere else) "less
important" changes when needed. Addresses problems b and c, problem a
still an issue.

* Instead of merging the full set of changes from another journal, we
could keep track of what the Oak client saved vs. what actually got
committed (after hook processing) and then during a merge try to
replay just those client changes before re-running the commit hooks. A
change set that fails because of merge conflicts or validation issues
could be moved to a separate conflict queue for later (possibly
manual) processing. This approach would solve all the above problems,
but could cause some surprises as the repository might occasionally
"undo" commits that previously passed without problems.

Given these tradeoffs I'm thinking that a default SegmentMK deployment
should start with just the root journal, and other journals (and their
merge behavior) should be configured depending on the requirements and
characteristics of particular deployment scenarios. We need to come up
with some better distributed test cases to determine what such
scenarios would look like and what the best journal configurations for
them would be.

BR,

Jukka Zitting

Mime
View raw message