lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Horatiu Lazu (JIRA)" <>
Subject [jira] [Created] (SOLR-12216) Add support for cross-cloud join
Date Thu, 12 Apr 2018 16:32:00 GMT
Horatiu Lazu created SOLR-12216:

             Summary: Add support for cross-cloud join 
                 Key: SOLR-12216
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
          Components: search
            Reporter: Horatiu Lazu

This patch is to propose the idea of extended the capabilities of the built-in join to allow
joining across SolrClouds. Similar to streaming's search function, the user can directly specify
the zkHost of the other SolrCloud and the rest of the syntax (from, to, fromIndex) can remain
the same. This join would be triggered when the zkHost parameter is specified, containing
the address of the other SolrCluster. It could also be packaged as a separate plugin.


In my testing, my current implementation is on average 4.5x faster than an equivalent streaming
expression intersecting from two search queries, one of which streams from another collection
on another SolrCloud. 
h5. How it works

Similar to the existing join, I created a QParser, but this join works as a post-filter. The
join first populates a hash set containing fields from the “from” index (i.e, the index
that’s not the one we’re running the query from). To obtain the fields, it establishes
a connection with the other SolrCloud using SolrJ through the ZooKeeper address specified,
and then uses a custom request handler that performs the query on the “from” index and
return back an array of strings containing a list of fields. Then, on the “to” index,
it iterates through the array sent as JavaBin and adds it to the hash set. After that, we
iterate through the NumericDocList for the “to” core’s join field, and if there’s
a value within the NumericDocList that’s found within our hash set, we collect it inside
the DelegatingCollector.

This allows for joining across sharded collections as well. 
h5. How I benchmarked

I created web-app that first reloads the collections, then sends 25 AJAX requests at once
to the Solr endpoint of varying query sizes (between 127 search results and 690,000), and
then recorded the results. After all responses are returned, the collection is reloaded, and
the equivalent streaming expressions are tested. This process is repeated 15 times, and the
average of the results is taken. 

Note: The first two requests are not counted in the statistics, because it “warms up”
the collection. For reference, after bouncing Solr and at least one query is executed, it
takes on average ~890ms for joining on two collections with about 690,000 results, while it
takes ~4.5 seconds using streaming expressions).


I have written unit tests written as well. I would appreciate some comments on this. Thank

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message