lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@googlemail.com>
Subject Regarding Transaction logging
Date Thu, 08 Sep 2011 09:35:39 GMT
hey folks,

we already have transaction logging on Solr side so I should have
started this discussion earlier. However, I want to bring this up to
the list since I think this is a very valuable feature also for plain
Lucene users and eventually this should also be available to them. I
don't think this needs to be a core feature at all but I think we need
to provide the necessary hooks in Lucene core to make this reliable
and consistent. I have a couple of concerns that which the current
extension mechanism we provide on the IndexWriter side this feature
can only be implemented in a sub-optimal way on the Solr (or basically
on top of lucene) but lemme elaborate this a little.

IndexWriter doesn't provide any transaction guarantees neither does it
give any guarantees on the order. So if you index two versions of a
document with the same delete key you can't tell which one wins unless
you prevent IW from seeing those two documents at the same time ie.
locking before you hit IW. This is basically what other implementation
do like ElasticSearch which uses locks assigned to buckets in an array
selected based on the del terms hash. However this gets a little more
complex once you get to DeleteQueries where you can't tell which
document is affected so they might be misplaced in the transaction log
if the order doesn't match the order the IW sees. Under the hood IW
does maintain such an order inside the DocumentsWriterDeleteQueue
which could be utilized to provide a total ordering that IMO should be
reflected in the transaction log.

Before I am going to propose ways of how this could be implemented I
want to check if other think we should provide more reliable ways for
users with the need for durability and consistent recovery.

simon

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message