lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: How to migrate content of a collection to a new collection
Date Thu, 24 Jul 2014 09:22:25 GMT
Thanks for replying

I tried this "poor mans" cursor approach out ad-hoc, but I get OOM. 
Pretty sure this is because you need all uniqueKey-values in FieldCache 
in order to be able to sort on it. We do not have memory for that - and 
never will. Our uniqueKey field is not DocValue.
Just out of curiosity
* Will I have the same OOM problem using the CURSOR-feature in later Solrs?
* Will the "poor mans" cursor approach still be efficient if my 
uniqueKey was DocValued, knowing that all values for uniqueKey (the 
DocValue file) cannot fit in memory (OS file cache)?

Regards, Per Steffensen

On 23/07/14 23:57, Chris Hostetter wrote:
> : billions of documents (not enough memory). Please note that we are on 4.4,
> : which does not contain the new CURSOR-feature. Please also note that speed is
> : an important factor for us.
>
> for situations where you know you will be processing every doc and order
> doesn't matter you can use a "poor mans" cursor by filtering on sccessive
> ranges of your uniqueKey field as described in the "Is There A
> Workaround?" section of this blog post...
>
> http://searchhub.org/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/
>
> * sort on uniqueKey
> * leave start=0 on every requets
> * add an fq to each request based on the last uniqueKey value from
>    the previous request.
>
>
> -Hoss
> http://www.lucidworks.com/
>


Mime
View raw message