lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gili Nachum <gilinac...@gmail.com>
Subject How can I get a monotonically increasing field value for docs?
Date Mon, 21 Sep 2015 06:34:51 GMT
I've implemented a custom solr2solr ongoing unidirectional replication
mechanism.

A Replicator (acting as solrJ client), crawls documents from SolrCloud1 and
writes them to SolrCloud2 in batches.
The replicator crawl logic is to read documents with a time greater/equale
to the time of the last replicated document.
Whenever a document is added/updated, I auto updated a a tdate field
"last_updated_in_solr" using TimestampUpdateProcessorFactory.

*My problem: *When a client indexes a batch of 100 documents, all 100 docs
have the same "last_updated_in_solr" value. This makes my ongoing
replication check for new documents to replicate much more complex than if
the time value was unique.

1. Can I use some other processor to generate increasing unique values?
2. Can I use the internal _version_ field for this? is it guaranteed to be
monotonically increasing for the entire collection or only per document,
with each add/update?
Any other options?

Schema.xml:
<field name="last_updated_in_solr" type="tdate" indexed="true"
stored="true" multiValued="false"/>

solrconfig.xml:
<updateRequestProcessorChain name="default">
       <processor class="solr.TimestampUpdateProcessorFactory">
           <str name="fieldName">last_updated_in_solr</str>
       </processor>
       <processor class="solr.LogUpdateProcessorFactory" />
       <processor class="solr.RunUpdateProcessorFactory" />
    </updateRequestProcessorChain>

I know there's work for a build-in replication mechanism, but it's not yet
released.
Using Solr 4.7.2.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message