lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Distributed Indexing
Date Mon, 14 Feb 2011 17:28:53 GMT
On Mon, Feb 14, 2011 at 10:04 AM, Alex Cowell <alxcwll@gmail.com> wrote:
> There seem to be some nuances which we have yet to encounter/discover like
> the way you've implemented the processCommit() method to wait for all the
> adds/deletes to complete before sending the commits. Are these things which
> you were aware of in advance that would need to be dealt with?

Yeah, it really just has to do with the fact that I was using multiple
threads to send update commands to other nodes.
This means that if you do an add and then a delete of the same doc,
those could get reordered to do the delete first, and then the add.
And the commit at the end could sneak in front of some adds and
deletes still in progress on other threads.

For true distributed indexing, I think we'll want a version number
somehow (perhaps based on timestamp by default) so updates can be
ordered, and all nodes can agree on the ordering.  For example, one
client could update node A with doc X, and a different client could
update node B with doc X.  If that happens very close together, we
need all shard replicas to agree which doc X will win.

-Yonik
http://lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message