lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: index update with database insertion
Date Mon, 21 Aug 2006 11:54:47 GMT
Jason Polites wrote:
> I'm not sure about the solution in the referenced thread.  It will work, 
> but
> doesn't it run the risk of breaching the transaction isolation of the
> database write?
> 
> The issue is when the index is notified of a database update.  If it is
> notified prior to the transaction commit, and the commit fails then you are
> out of synch.  If it is notified after commit, and the notify fails you are
> out of sync.
> 
> Maybe more of a "publish/subscribe" model would work (or.. maybe this is
> what was suggested and I am way off!).  That is, when the DB write occurs,
> and within the same transaction, an "index" task is published to some new
> "IndexTask" table which your index process polls.  This way if the index
> process is offline.. when it comes back up it simply has a list of tasks to
> perform.
> 
> Apologies if this is what you meant Michael.
> 
> Or.. better still (for transaction isolation, but arguably not performance)
> do the index update within the same transaction.  I have just been
> implementing some test code using Compass (Open Symphony) which provides
> full transaction isolation for Lucene.

That is an excellent point Jason -- that thread I linked to doesn't
talk to how to have transaction isolation with your DB updates.

I like the idea of a separate "IndexTasks" table.  This is better
than a simple timestamp column added to all content tables because it
can handle deleted docs correctly.  And it allows full decoupling of
when the DB transaction is committed vs. when Lucene commits its
corresponding index updates.

I think ideally you would indeed want to commit both at once (your
last paragraph above), meaning, your end users see changes to DB
simultaneous with changes to searching on the Lucene index.  Compass
sounds like it will address this nicely!

Solr also adds some transactional semantics on top of Lucene, in that
you add/delete documents, but these changes are not visible to the end
user until a "commit" is issued.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message