jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jukka Zitting" <jukka.zitt...@gmail.com>
Subject Re: Next Generation Persistence
Date Mon, 11 Jun 2007 18:28:02 GMT
Hi,

On 6/11/07, Thomas Mueller <thomas.tom.mueller@gmail.com> wrote:
> It sounds like MVCC (multi version concurrency control) in databases.

Yes, MVCC was one of my inspirations (see [1]), although I must admit
that I only know MVCC on a high level.

[1] http://article.gmane.org/gmane.comp.apache.jackrabbit.devel/10642/match=mvcc

> Question: in your view, do we need to change the PersistenceManager
> API as well?

I think that's inevitable.

> * Removing nodes *
> Last section: when removing B, the child C is removed as well. Is it
> important to say 'remove C', and not 'remove B including children' in
> the revision?

I think the essential point to consider is the referenceability of the
nodes. We probably need to explicitly mark all referenceable nodes as
deleted, but I think other nodes can be handled just by marking the
root of the non-referenceable subtree as deleted.

> What would happen if another session would add a child D
> to C in the meantime, and commit this change?

During commit the other session would notice that intervening changes
have occurred and would update the local changes to match the latest
state of the repository. This update process should validate paths at
least up to the closest referenceable ancestor node.

> If there is no locking, how to do large deletes / updates?

My idea is to have all synchronization at the point when transient
changes are persisted. See the flowchart in the "Draft revision"
section.

The time required to validate a large delete/update could become quite
high and such changes could easily end up invalidating the transient
changes of a number of concurrent sessions, but I guess similar
concerns apply regardless of how large changes are handled.

> * Revision scope *
> In my opinion, the scope should be the entire repository.

I tend to agree after considering the alternatives.

> * Base Revision *
> The text talks about 'the base revision of the session' (can be
> updated in some cases). But to support  Item.refresh, a session needs
> to keep multiple base revisions: one for the session, and one for each
> refreshed Item (including children)?

The spec doesn't specify whether or when the session state gets
refreshed so I think we could implement Item.refresh like this:

    public void refresh(boolean keepChanges) throws ...{
        if (!keepChanges) {
            // drop transient changes for this item and descendants
        }
        getSession().refresh(true);
    }

This way we could keep a single consistent base revision for the entire session.

> > The base revision of a session can optionally be changed when more
> > recent revisions are persisted during the session lifetime.
> Does 'optionally' mean it is a setting of the session?

The "auto-refresh" mode could be set either for the entire repository
or for an individual session.

> Is there a JCR API feature to set this option?

No, the spec doesn't really specify when or even if session state gets
refreshed between explicit refresh() calls.

We could for example add a custom JackrabbitSession.setAutoRefresh()
method or allow the option to be specified during login like this:

    Session session = repository.login("workspace?refresh=auto");

> * Persisting a Subtree *
> > If the operation fails, then the two new revisions are discarded and
> > no changes are made to the session.
> Item.save says: 'If validation fails, then no pending changes are
> saved and they remain recorded on the Session'

My intention was that the original state of the session, including the
original draft revision, is not changed if the operation fails. To
clarify:

Let's represent the global state of the repository as R = R(X), where
X is the last persisted revision, and the local state of a session as
S = S(B, D), where B is the base revision and D the draft revision of
the session.

Now, say that D contains subtree operations P and some other changes
Q, i.e. D = P + Q and intersect(P, Q) = 0. Then persisting the subtree
operations would result in state changes R' = R(X + P) and S' = S(X +
P, Q) assuming that the update operation B => X and subsequent
validation of P succeeds. If the operation fails, then the session
state isn't changed, S' = S = S(B, D).

I hope I didn't get too complex there...

> * Workspace Operations *
> > If the operation succeeds, the session is updated to use the persisted
> > revision as the new base revision.
> I think the base revision of the session should not be updated in this case.

I think it would be very confusing for a client to do a workspace move
or a copy and then not to be able to see the changes before an
explicit refresh.

As discussed for Item.refresh, I think we have quite a lot of freedom
in deciding when and how the session state gets refreshed.

> * Transactions *
> > This model can also easily support two-phase commits in a distributed
> > transaction.
> I agree, two-phase commit is no problem. However XAResource.recover
> (obtains the list of prepared transactions) is tricky to implement. As
> far as I know, is currently not supported. If we want it in the
> future, we better think early how to implement it. I suggest we
> describe the problem and possible solutions, but wait with the
> implementation until required.

You're right, we currently have nothing like that. I definitely don't
understand all the fine points in XA, but assuming we keep all
relevant information persisted within a revision I think it should be
possible to implement fairly complex recovery mechanisms.

> * Namespace and Node Type Management *
> Namespace and node type management in jcr:system: Good idea! However
> without custom data structures it will be slow (if jcr:system subtree
> is read whenever a namespace is resolved). What about custom data
> structures that cache the latest state (and listen for changes to the
> relevant subtree)? I guess only the latest version is relevant, or is
> there a situation where older versions of node types / namespaces are
> required? If yes, things will be complicated.

Caching the information in custom data structures would definitely
make sense. We even persist those structures as a part of the revision
whenever namespace or node type changes are made, and perhaps include
a reference to the last revision where such "schema" changes were made
in each persisted revision. A session could then access the latest
cache simply by following that reference. This way we wouldn't need
any observation listeners and we could support concurrent repository
views with different schema versions.

> * Internal Data Structures *
> I don't fully understand this section. I think this section should be
> extended.

Agreed. :-) I have some vague ideas on how we could best persist and
access the revisions data, but we're still far from even working
prototypes. I included the data structure section in the proposal
mostly to record some of my early ideas and to make it clear that this
is probably the area that will need most work...

Having someone with solid database background commenting is much
appreciated. :-)

> In databases, the approach is usually:
>
> A: There is a main store (where the 'base' revision of items are kept,
> indexed by item).
> B: Committed revisions are stored sequentially in the redo log. Can
> only be read sequentially.
> C: Draft and old revisions are mainly kept in-memory (saved to disk
> only if no space).
> D: Each session keeps an undo log for uncommitted changes.
> E: Committed revisions are persisted in the main store, and if session
> references an older revision of the same item, an in-memory copy is
> made (copy on write, see C).
>
> Of course we don't need to rebuild a database, but maybe reuse some ideas.

Exactly! I would be very interested in exploring ways to somehow merge
the best practices from the row-based database world and the way for
example Subversion
incrementally stores a tree structure. See also Tobias' notes on this.

Again, I must confess that I'm not the best expert in data structures
for low-level persistence as I've spent most of my career working
higher up on the software stack, mostly building stuff on top of SQL,
HTTP, and nowadays JCR.

> * Combined Revisions *
> I don't understand why to do that, but probably because you have
> different internal data structures in mind.

I'm coming from a high-end view where there are no globally indexed
set of "base" versions of the stored items, and content is instead
represented as a sequential set of revision that each contain a change
set against previous revisions. In systems like Subversion such
storage model works fine as everything is accessed hierarchically, but
as soon as referenceability (with back-references!) and search
features are included I fear there will be problems with such an
approach. The main motivation for combining (old) revisions is to
improve the efficiency of indexing.

> When to send large objects to the server: There are two use cases:
> - Client is far away: keep changes on the client as long as possible,
> send batches
> - Client is close by: avoid temporary copies of large (binary) data on
> the client
> Both cases should be supported, but which one is more important?

Agreed. I think we can (at least for now) focus on a local client as
the SPI work is already trying to solve the remote access and batching
mechanisms. I guess we eventually need to produce some sort of
synthesis of SPI and NGP (or another future persistence architecture),
but I guess that would be something for Jackrabbit 4.0 or 5.0...

BR,

Jukka Zitting

Mime
View raw message