lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Streaming large updates with SolrJ
Date Thu, 03 Jul 2014 02:14:13 GMT

: Now that I think about it, though, is there a way to use the Update Xml
: messages with something akin to the cloud solr server?  I only see examples
: posting to actual Solr instances, but we really need to be able to take
: advantage of the zookeepers to send our updates to the appropriate servers.

Part of your confusion may be that there are 2 different way of leveraging
the "SolrServer" APIs (either CloudSolrServer, or any other SolrServer 
implementation)...

 * syntactic sugar apis like "SolrServer.add(...)" which require
SolrInputDocuments
 * the lower level methods like SolrRequest.process(solrServer)

...with the later, you can subclass AbstractUpdateRequest and implement 
getContentStreams() to send whatever (lazy constructed) stream of bytes 
you want to Solr.

Altenatively: you could conider subclassing SolrInputField with something 
thta knows how to lazy fetch the data you want to stream across the wire, 
and then (unless i'm missing something?) you can still use the sugar APIs 
with SolrInputDocuments but only individual field values will need to 
exist in RAM at any one time (as the BinaryWriter or XmlWriter calls 
SolrInputField.getValues() on your custom class to stream over the wire)

However: if you are using SolrCloud, none of this will help you work 
arround the previuosly mentioned SOLR-6199, which affects how much RAM 
Solr needs to use on the server side when forwarding docs arround to 
replicas.



-Hoss
http://www.lucidworks.com/

Mime
View raw message