lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrian Tarau (JIRA)" <>
Subject [jira] Commented: (LUCENE-1149) add XA transaction support
Date Tue, 17 Jun 2008 20:09:45 GMT


Adrian Tarau commented on LUCENE-1149:

I had the same problem, so one year ago I implemented transactions on top of Lucene(I had
an idea how to do it, but I also peeked a little bit in Compass sources). Basically I create
a new index every time when a new transaction is started and this new index is made visible
if commit is successful. Of course, you will still need to refresh your IndexReader(s). I
planned also for a form of two-phase commit but right now it works pretty well doing everything
manually - controlling when to commit/rollback/restore.

How it works :

1. start 2 transactions, one with MySQL one with Lucene(I have an abstract layer on top of
both engines so the API calls are the same), commit first Lucene.
2. if Lucene fails, rollback MySQL.
3. if Lucene commit(precommit actually - everything is stored, I have a new optimized index,
ready to be used) is successful, commit MySQL.
4. if MySQL fails, "cancel" Lucene transaction(remove index). There is a slightly chance to
not being able to cancel the Lucene transaction(which is a new index). This index will hang
there on disk/memory until the session is closed or the application is restarted.
5. if MySQL commit is successful, the newly created Lucene index is "activated". There is
a slightly chance to not being able to "activate" Lucene transaction(application crashes,
disk full so cannot write state on disk) - so in this case the global transaction is "in-doubt".
In this case a system administrator must act and activate the index(the index is fine, but
not active) - after he fixes any existing problems - or decided to clear the database(you
need eventually a unique global transaction id stored in the database too, as an additional
column). Your data is in the database, the application works fine but some entries are "invisible"
when searching.

So is kind of a two-phase commit. 

> add XA transaction support
> --------------------------
>                 Key: LUCENE-1149
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: robert engels
> Need to add XA transaction support to Lucene.
> Without XA support, it is difficult to keep disparate resources (e.g. database) in sync
with the Lucene index.
> A review of the XA support added to Hibernate might be a good start (although Hibernate
almost always uses a XA capable backing store database).
> It would be ideal to have a combined IndexReaderWriter instance, then create a XAIndexReaderWriter
which wraps it.
> The implementation might be as simple as a XA log file which lists the XA transaction
id, and the segments XXX number(s), since Lucene already allows you to rollback to a previous
version (??? for sure, or does it only allow you to abort the current commit).
> If operating under a XA transaction, the no explicit commits or rollbacks should be allowed
on the instance.
> The index would be committed during XA prepare(), and then if needed rolledback when
requested. The XA commit() would be a no-op.
> There is a lot more to this but this should get the ball rolling.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message