lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: tipping point for using solrcloud—or not?
Date Mon, 02 Oct 2017 22:39:05 GMT
Short form: Use SolCloud from what you've described.

NRT and M/S is simply oil and water. The _very_ best you can do when
searching slaves is
master's commit interval + slave polling interval + time to transmit
the index to the slave + autowarming time on the slave.

Now, that said, when you say NRT it's really "10 minutes is OK" then
M/S will work for you.

But otherwise I'd be using SolrCloud.

Best,
Erick

On Mon, Oct 2, 2017 at 1:48 PM, John Blythe <johnblythe@gmail.com> wrote:
> thanks for the responses, guys.
>
> erick: we do need NRT in several cases. also in need of HA pending where
> the line is drawn. we do need it relatively speaking, i.e. w/i our user
> base. if the largest of our cores falters then our business is completely
> stopped till we can get everything reindexed.
>
> is there a general rule when it comes to query rate and efficiency between
> Cloud and M/S? in either case we need to add complexity to the system so,
> if it's a jump ball, that will be the thing that likely tips in favor.
>
> emir: the logs aren't write intensive. what are the core benefits to
> splitting up the machine if there isn't a jvm load issue we're currently
> experiencing?
>
> i can def provide more info that could help in the discussion. help me know
> the best way / stuff to send if you can please.
>
> thanks again for the help guys-
>
> --
> John Blythe
>
> On Fri, Sep 29, 2017 at 10:27 AM, Erick Erickson <erickerickson@gmail.com>
> wrote:
>
>> SolrCloud. SolrCloud. SolrCloud.
>>
>> Well, it actually depends. I recommend people go to SolrCloud when any
>> of the following apply:
>>
>> > The instant you need to break any collection up into shards because
>> you're running into the constraints of your hardware (you can't just keep
>> adding memory to the JVM forever).
>>
>> > You need NRT searching and need multiple replicas for either your
>> traffic rate or HA purposes.
>>
>> > You find yourself dealing with lots of administrative complexity for
>> various indexes. You have what sounds like 6-10 cores laying around. You
>> can move them to different machines without going to SolrCloud, but then
>> something has to keep track of where they all are and route requests
>> appropriately. If that gets onerous, SolrCloud will simplify it.
>>
>> If none of the above apply, master/slave is just fine. Since you can
>> rebuild in a couple of hours, most of the difficulty with M/S when the
>> master goes down are manageable. With a master and several slaves, you
>> have HA, and a load balancer will see to it that some are used.
>> There's no real need to exclusively search on the slaves, I've seen
>> situations where the master is used for queries as well as indexing.
>>
>> To increase your query rate, you can just add more slaves to the hot
>> index, assuming you're content with the latency between indexing and
>> being able to search newly indexed documents.
>>
>> SolrCloud, of course, comes with the added complexity of ZooKeeper.
>>
>> Best,
>> Erick
>>
>>
>>
>> On Fri, Sep 29, 2017 at 5:34 AM, John Blythe <johnblythe@gmail.com> wrote:
>> > hi all.
>> >
>> > complete noob as to solrcloud here. almost-non-noob on solr in general.
>> >
>> > we're experiencing growing pains in our data and am thinking through
>> moving
>> > to solrcloud as a result. i'm hoping to find out if it seems like a good
>> > strategy or if we need to get other areas of interest handled first
>> before
>> > introducing new complexities.
>> >
>> > here's a rundown of things:
>> > - we are on a 30g ram aws instance
>> > - we have ~30g tucked away in the ../solr/server/ dir
>> > - our largest core is 6.8g w/ ~25 segments at any given time. this is
>> also
>> > the core that our business directly runs off of, users interact with,
>> etc.
>> > - 5g is for a logs type of dataset that analytics can be built off of to
>> > help inform the primary core above
>> > - 3g are taken up by 3 different third party sources that we use solr to
>> > warehouse and have available for query for the sake of linking items in
>> our
>> > primary core to these cores for data enrichment
>> > - several others take up < 1g each
>> > - and then we have dev- and demo- flavors for some of these
>> >
>> > we had been operating on a 16gb machine till a few weeks ago (actually
>> > bumped while at lucene revolution bc i hadn't noticed how much we'd
>> > outgrown the cache size's needs till the week before!). the load when
>> doing
>> > an import or running our heavier operations is much better and doesn't
>> fall
>> > under the weight of the operations like it had been doing.
>> >
>> > we have no master/slave replica. all of our data is 'replicated' by the
>> > fact that it exists in mysql. if solr were to go down it'd be a nice big
>> > fire but one we could recover from within a couple hours by simply
>> > reimporting.
>> >
>> > i'd like to have a more sophisticated set up in place for fault tolerance
>> > than that, of course. i'd also like to see our heavy, many-query based
>> > operations be speedier and better capable of handling multi-threaded runs
>> > at once w/ ease.
>> >
>> > is this a matter of getting still more ram on the machine? cpus for
>> faster
>> > processing? splitting up the read/write operations between master/slave?
>> > going full steam into a solrcloud configuration?
>> >
>> > one more note. per discussion at the conference i'm combing through our
>> > configs to make sure we trim any fat we can. also wanting to get
>> > optimization scheduled more regularly to help out w segmentation and
>> > garbage heap. not sure how far those two alone will get us, though.
>> >
>> > thanks for any thoughts!
>> >
>> > --
>> > John Blythe
>>

Mime
View raw message