lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohit Jain <rohit.j...@esgyn.com>
Subject RE: Parallel API interface into SOLR
Date Mon, 12 Jun 2017 17:11:50 GMT
Erick,

I think so, although I may have overlooked something.  The idea is that we would make a request
to the API from a single client but expect multiple streams of results to be returned in parallel
to multiple parallel processes that we have set up to receive those results from SOLR.  Do
these interfaces provide that?  This has always been the issue with interfaces like JDBC /
ODBC as well, since they don't provide a mechanism to consume the results in parallel streams.
 There is no protocol set up to do that.  I was just wondering if there was for SOLR and what
would be an example of that.

Rohit

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Monday, June 12, 2017 11:56 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Parallel API interface into SOLR

Have you looked at Streaming Aggregation/Streaming Expressions/Parallel SQL etc?

Best,
Erick

On Mon, Jun 12, 2017 at 9:24 AM, Rohit Jain <rohit.jain@esgyn.com> wrote:
> Hi folks,
>
> We have a solution where we would like to connect to SOLR via an API, submit a query,
and then pre-process the results before we return the results to our users.  However, in some
cases, it is possible that the results being returned by SOLR, in a large distributed cluster
deployment, is very large.  In these cases, we would like to set up parallel streams, so that
each parallel SOLR worker feeds directly into one of our processes distributed across the
cluster.  That way, we can pre-process those results in parallel, before we consolidate (and
potentially reduce / aggregate) the results further for the user, who has a single client
connection to our solution.  Sort of a MapReduce type scenario where our processors are the
reducers.  We could consume the results as returned by these SOLR Worker processes, or perhaps
have them shuffled based on a shard key, before our processes would receive them.
>
> Any ideas on how this could be done?
>
> Rohit Jain
Mime
View raw message