jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miro Walker" <miro.wal...@cognifide.com>
Subject RE: Google Summer of Code project for Jackrabbit
Date Fri, 26 May 2006 00:30:43 GMT
Nicolas,

Hm - ok. I think you would need to ensure that you don't just serialize
out to a local filesystem (as this in itself is not a transactional
storage, and so does not guarantee integrity of the serializations).
>From my perspective, the system could only be usable if it were to be
configurable to write out to the same transactional storage (i.e.
database instance) as the rest of the repository data, so that if after
failure it is necessary to restart on different hardware, there are no
issues with locally stored temporary files.

If this can be guaranteed, then I see no reason why completing the write
of the changelog after restart shouldn't work.

With regard to your other question, the key point is that for
transactions in jackrabbit to mean anything, the underlying storage
mechanism must also be transactional, and the commit method must rely on
the successful completion of a commit to the underlying storage. If this
is the case, then an exception during commit would simply cause a
rollback of the underlying storage, leaving the system intact.
Obviously, the key point would be that the operation of removing an
entry from the changelog and writing it out to the PM would have to
happen within the same transaction (i.e. it must be impossible to have
the write to the PM commit without the changelog being updated
accordingly or vice versa). 

miro

-----Original Message-----
From: Nicolas Toper [mailto:ntoper@gmail.com] 
Sent: 26 May 2006 01:18
To: dev@jackrabbit.apache.org
Subject: Re: Google Summer of Code project for Jackrabbit

Hi Miro,

I completely agree with you. I already detected the issue. I propose to
serialize in the FS (or use another PM in a specific workspace) each
ChangeLog object and if the Jackrabbit is crashing, I would commit them
after startup anyway. Then we would still have transactionnal integrity.
It
is just a delayed write.

Is writing a ChangeLog object after a restart of the server something
that
could work?

My issue is more: I save the ChangeLog object, but at the "real" commit
time, an exception might still occur. How to handle it? Cf. in my last
mail:
"I can even make persistant the modification using an already existing
PM in
case of crash. One question though: I cannot guarantee the transaction
would
be later committed without exception. We can choose to ignore this issue
or
add an asynchronous way to warn the session. What are your thoughts on
this?
"

Would this strategy work? Another advantage of this strategy is: it
might
allow us later to add serializable transaction through a 2 phase locking
for
instance (since we would have a layer to delay a transaction if needed).

nico

On 5/26/06, Miro Walker <miro.walker@cognifide.com> wrote:
>
> Hi Nicolas,
>
> I'm a little concerned about the transactional integrity of a system
> using the proxy PM approach. If the system were to fail (out of
memory,
> hardware failure, etc.) during a backup, it would appear that we could
> lose all data that is cached by the proxy PM, even though these would
> represent "committed" transactions. Unless I've misunderstood, this
> would not be acceptable in any scenario where transaction support is
> mandated.
>
> In addition, even if a backup is not running, would such a
modification
> to the core of the jackrabbit system compromise the transactional
> integrity of the overall system? If a commit could complete without
the
> data having been persisted to the underlying datastore (as it is still
> being buffered by the proxy PM), then from my perspective this would
be
> taking jackrabbit backwards in terms of reliability.
>
> I'm also somewhat nervous about such modifications to the core, as it
> opens up the door to some nasty regressions for the system as a whole,
> and would need some serious testing in terms of threading &
concurrency,
> transactionality, etc.
>
> What do you think?
>
> Miro
>
> -----Original Message-----
> From: Nicolas Toper [mailto:ntoper@gmail.com]
> Sent: 25 May 2006 22:12
> To: dev@jackrabbit.apache.org
> Subject: Re: Google Summer of Code project for Jackrabbit
>
> Just one last thing: if we agree on this, I will update the
> specification
> document and post it back to the ML (and on my blog) before I start
> coding.
> It seems we can reach a consensus soon.
>
> On 5/25/06, Nicolas Toper <ntoper@gmail.com> wrote:
> >
> > Just to summarize everything we have said on this issue.
> >
> > There are two kinds of lock: the jcr.Lock and the
> > EDU.oswego.cs.dl.util.concurrent.*. The two are somewhat not
related.
> Am I
> > correct?
> >
> > There are no issues with jcr.Lock (we can still read a node).
> >
> > We need some mutex to avoid inconsistant IO operations using the
> > util.concurrent package. I like Tobias approach to add a proxyPM. It
> > seems easy. But is this solution elegant enough and maintenable in
the
> long
> > run? Would it help us later? (I think so since it would allow
delayed
> write
> > which open the way for a 2 phase locking algorithm.) I am not in
this
> > project since long enough to judge :p
> >
> > Why didn't Jackrabbit go for serializable transaction by the way? I
> have
> > checked the code and it seems we have all the needed kind of locks
to
> > support 2PL (out of scope of the current project of course).
> >
> > If we plan to support serializable transaction soon, then case 2 is
> > acceptable. Is this the case?
> >
> > About Tobias ProxyPM: I am ok to write it although it is out of
scope
> of
> > the initial project, you all seem to really need it, so let's go for
> it.
> > Jukka?
> >
> > For a specific workspace, I would still allow read operations from
> other
> > sessions and isolate all write access (this way there will be no
> conflict).
> > I can even make persistant the modification using an already
existing
> PM in
> > case of crash. One question though: I cannot guarantee the
transaction
> would
> > be later committed without exception. We can choose to ignore this
> issue or
> > add an asynchronous way to warn the session. What are your thoughts
on
> this?
> >
> >
> > This means a modification in the core package. Are you all OK with
> this?
> >
> >
> > By the way, this kind of algorithm is called a pessismistic receiver
> based
> > logging message algorithm. We use it in distributed systems.
> >
> >
> >
> > Thanks for your support and ideas.
> >
> > nico
> > My blog! http://www.deviant-abstraction.net !!
> >
> >
> >
> >
> > On 5/25/06, Tobias Bocanegra < tobias.bocanegra@day.com> wrote:
> > >
> > > i think there is a consensus of what backup levels there can be:
> > >
> > > 1) read locked workspaces
> > > 2) write locked workspaces
> > > 3) hot-backup (i.e. "SERIALIZABLE" isolation)
> > >
> > > in case 1, the entire workspace is completely locked (rw) and no
one
> > > else than the backup-session can read-access the workspace. this
is
> > > probably the easiest to implement and the least desirable.
> > >
> > > in case 2, the entire workspace becomes read-only, i.e. is
> > > write-locked. so all other sessions can continue reading the
> > > workspace, but are not allowed to write to it. this is also more
or
> > > less easy to implement, intoducing a 'global' lock in the lock
> > > manager.
> > >
> > > in case 3, all sessions can continue operations on the workspace,
> but
> > > the backup-session sees a snapshot view of the workspace. this
would
> > > be easy, if we had serializable isolated transactions, which we
> don't
> > > :-(
> > >
> > > for larger productive environments, only case 3 is acceptable. the
> way
> > > i see of how to impement this, is to create a
> > > 'proxy-persistencemanager' that sits between the
> > > shareditemstatemanager and the real persistencemanager. during
> normal
> > > operation, it just passes the changes down to the real pm, but in
> > > backup-mode, it keeps an own storage for the changes that occurr
> > > during the backup. when backup is finished, it resends all changes
> > > down to the real pm. using this mechanism, you have a stable
> snapshot
> > > of the states in the real pm during backup mode. the export would
> then
> > > access directly the real pm.
> > >
> > > regards, toby
> > >
> > >
> > > On 5/25/06, Nicolas Toper < ntoper@gmail.com> wrote:
> > > > Hi David,
> > > >
> > > > Sorry to have been unclear.
> > > >
> > > > What I meant is we have two different kinds of backup to
perform.
> > > >
> > > > In one use case I call "regular backup", it is the kind of
backup
> you
> > > > perform every night. You do not care not to grab the content
just
> > > updated,
> > > > since you will have it the day after.
> > > >
> > > > In the other use case I call "exceptional backup", you want to
> have
> > > all the
> > > > data because for instance you will destroy the repository
> afterwards.
> > > >
> > > > Those two differs I think in small points. For instance, for
> "regular
> > > > backup", we don't care about transaction started but not
> committed. In
> > > the
> > > > second one, we do.
> > > >
> > > > I propose to support only the first use case. The second one
would
> be
> > > added
> > > > easily later.
> > > >
> > > > I don't know how JackRabbit is used in production environment.
Is
> it
> > > > feasible to lock workspace once at a time or it is too
cumbersome
> for
> > > the
> > > > customer?
> > > >
> > > > For instance, if backuping a workspace needs a two minutes
> workspace
> > > > locking, then it can be done without affecting availibility (but
> it
> > > would
> > > > affect reliability). We need data to estimate if it is needed.
Can
> you
> > > give
> > > > me the size of a typical workspace please?
> > > >
> > > > I am OK to record the transaction and commit it after the
locking
> has
> > > > occured but this means changing the semantic of Jackrabbit (a
> > > transaction
> > > > initiated when a lock is on would be performed after the lock is
> > > released
> > > > instead of raising an exception ) and I am not sure everybody
> would
> > > think it
> > > > is a good idea. We would need to add a transaction log (is there
> one
> > > > already?) and parse transaction to detect conflict (or capture
> > > exception
> > > > maybe). We would not be able to guarantee anymore a transaction
is
> > > > persistent and it might have an impact on performance. And what
> about
> > > time
> > > > out when running a transaction?
> > > >
> > > > Another idea would be: monitor Jackrabbit and launch the backup
> when
> > > we have
> > > > a high probability no transaction are going to be started. But I
> think
> > > > sysadmin already know when load is minimal on their system.
> > > >
> > > > Another idea would be as Miro stated, use more "lower" level
> strategy
> > > > (working on the DB level or directly on the FS). It was actually
> my
> > > first
> > > > backup strategy but Jukka thought have to be able to use the
tool
> to
> > > migrate
> > > > from one PM to another
> > > >
> > > > Here is my suggestion on the locking strategy: we can extend the
> > > backup tool
> > > > later if needed. Right now even with a global lock, it is an
> > > improvement
> > > > compared to the current situation. And I need to release the
> project
> > > before
> > > > August 21.
> > > >
> > > > I would prefer to start with locking one workspace at a time and
> if I
> > > have
> > > > still time then find a way to work with minimal lock. I will
> > > most  probably
> > > > keep working on Jackrabbit after the Google SoC is over. Are you
> OK
> > > with
> > > > this approach?
> > > >
> > > > We are OK on the restore operation. Good idea for the replace or
> > > ignore
> > > > option but I would recommend to build it only for existing nodes
> :p
> > > > Properties might be more difficult to handle and not as useful
> (and it
> > > > raises a lot more questions).
> > > >
> > > > nico
> > > > My blog! http://www.deviant-abstraction.net !!
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
> --
> a+
> nico
> My blog! http://www.deviant-abstraction.net !!
>



-- 
a+
nico
My blog! http://www.deviant-abstraction.net !!

Mime
View raw message