lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Re: Streaming large updates with SolrJ
Date Thu, 03 Jul 2014 14:28:14 GMT
Thanks Hoss, that's a good explanation.  I don't have much experience with
the non-sugar parts of the API, so this was a good summary.  I suppose I
can at least help out the client heap size this way.


On Wed, Jul 2, 2014 at 10:14 PM, Chris Hostetter <hossman_lucene@fucit.org>
wrote:

>
> : Now that I think about it, though, is there a way to use the Update Xml
> : messages with something akin to the cloud solr server?  I only see
> examples
> : posting to actual Solr instances, but we really need to be able to take
> : advantage of the zookeepers to send our updates to the appropriate
> servers.
>
> Part of your confusion may be that there are 2 different way of leveraging
> the "SolrServer" APIs (either CloudSolrServer, or any other SolrServer
> implementation)...
>
>  * syntactic sugar apis like "SolrServer.add(...)" which require
> SolrInputDocuments
>  * the lower level methods like SolrRequest.process(solrServer)
>
> ...with the later, you can subclass AbstractUpdateRequest and implement
> getContentStreams() to send whatever (lazy constructed) stream of bytes
> you want to Solr.
>
> Altenatively: you could conider subclassing SolrInputField with something
> thta knows how to lazy fetch the data you want to stream across the wire,
> and then (unless i'm missing something?) you can still use the sugar APIs
> with SolrInputDocuments but only individual field values will need to
> exist in RAM at any one time (as the BinaryWriter or XmlWriter calls
> SolrInputField.getValues() on your custom class to stream over the wire)
>
> However: if you are using SolrCloud, none of this will help you work
> arround the previuosly mentioned SOLR-6199, which affects how much RAM
> Solr needs to use on the server side when forwarding docs arround to
> replicas.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message