lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetas Joshi <chetas.jo...@gmail.com>
Subject Parallelize Cursor approach
Date Sat, 05 Nov 2016 00:48:37 GMT
Hi,

I am using the cursor approach to fetch results from Solr (5.5.0). Most of
my queries return millions of results. Is there a way I can read the pages
in parallel? Is there a way I can get all the cursors well in advance?

Let's say my query returns 2M documents and I have set rows=100,000.
Can I have multiple threads iterating over different pages like
Thread1 -> docs 1 to 100K
Thread2 -> docs 101K to 200K
......
......

for this to happen, can I get all the cursorMarks for a given query so that
I can leverage the following code in parallel

cursorQ.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark)
val rsp: QueryResponse = c.query(cursorQ)

Thank you,
Chetas.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message