lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Pendlebury <greg.pendleb...@gmail.com>
Subject Re: Deep paging in parallel with solr cloud - OutOfMemory
Date Mon, 17 Mar 2014 23:54:19 GMT
Sorry, I meant one thread requesting records 1 - 1000, whilst the next
thread requests 1001 - 2000 from the same ordered result set. We've
observed several of our customers trying to harvest our data with
multi-threaded scripts that work like this. I thought it would not work
using cursor marks... but:

A) I could be wrong, and
B) I could be talking about parallel in a different way to Mike.

Ta,
Greg



On 18 March 2014 10:24, Yonik Seeley <yonik@heliosearch.com> wrote:

> On Mon, Mar 17, 2014 at 7:14 PM, Greg Pendlebury
> <greg.pendlebury@gmail.com> wrote:
> > My suspicion is that it won't work in parallel
>
> Deep paging with cursorMark does work with distributed search
> (assuming that's what you meant by "parallel"... querying sub-shards
> in parallel?).
>
> -Yonik
> http://heliosearch.org - solve Solr GC pauses with off-heap filters
> and fieldcache
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message