lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laurence Rowe" <laure...@lrowe.co.uk>
Subject Re: Ordering updates
Date Thu, 04 Dec 2008 21:12:21 GMT
Hi,

We currently have a single Solr server, with a single index. There are
a number of CMS processes distributed over a number of servers, with
each CMS process sending an update to the Solr index when changes are
made to a content object.

My concern is that a scenario is possible where a content object is
changed and reindexed concurrently by two CMS processes. The database
ensures consistency within the CMS, these transactions get comitted as
T1 and T2. But I cannot see how to ensure that the reindexing
operations (that result in a delete and add for the document) are
processed in the order R1 then R2, rather than R2 then R1. In the
second case the index record is now inconsistent with the content
object in the database.

I would like to supply a transaction id with the reindex request, and
configure Solr such that a reindex operation is processed if and only
if the supplied transaction id is greater than the currently indexed
transaction id.

Otherwise the only way I can see to guarantee consistency is 1) have
index operations processed by a single writer, or 2) commit the index
operation between database prepare and commit statements.

The first is not desirable as we introduce a single point of failure
(in addition to the single Solr server) and delay updating the index.
The second is not desirable because it reduces the throughput of the
database, and with a distributed Solr setup would not solve the
problem.

>From what I can tell this conditional indexing feature is not
supported by Solr. Might it be supported by Lucene but not exposed by
Solr?

Thanks,

Laurence

2008/12/4 Shalin Shekhar Mangar <shalinmangar@gmail.com>:
> It is not clear how you are using Solr i.e. distributed vs single index.
>
> Summarily, Solr does not update documents. It overwrites the old document
> with the new one if an old document with the same uniqueKey exists in the
> index.
>
> Does that answer your question?
>
> On Thu, Dec 4, 2008 at 1:46 AM, Laurence Rowe <l@lrowe.co.uk> wrote:
>
>> Hi,
>>
>> Our CMS is distributed over a cluster and I was wandering how I can
>> ensure that index records of newer versions of documents are never
>> overwritten by older ones. Amazon AWS uses a timestamp on requests to
>> ensure 'eventual consistency' of operations. Is there a way to supply
>> a transaction ID with an update so an update is conditional on the
>> supplied transaction id being greater than the existing indexed
>> transaction id?
>>
>> Laurence
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Mime
View raw message