lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: How to migrate content of a collection to a new collection
Date Wed, 23 Jul 2014 21:57:12 GMT

: billions of documents (not enough memory). Please note that we are on 4.4,
: which does not contain the new CURSOR-feature. Please also note that speed is
: an important factor for us.

for situations where you know you will be processing every doc and order 
doesn't matter you can use a "poor mans" cursor by filtering on sccessive 
ranges of your uniqueKey field as described in the "Is There A 
Workaround?" section of this blog post...

http://searchhub.org/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/

* sort on uniqueKey
* leave start=0 on every requets
* add an fq to each request based on the last uniqueKey value from 
  the previous request.


-Hoss
http://www.lucidworks.com/

Mime
View raw message