lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pushkar Raste <pushkar.ra...@gmail.com>
Subject Re: Spread SolrCloud across two locations
Date Fri, 26 May 2017 14:07:16 GMT
Damn,
Math is hard

DC1 : 3 non observers
DC2 : 2 non observers

3 + 2 = 5 non observers

Observers don't participate in voting = non observers participate in voting

5 non observers = 5 votes

In addition to the 2 non observer, DC2 also has an observer, which as you
pointed out does not participate in the voting.

We still have 5 voting nodes.


Think of the observer as a standby name node in Hadoop 1.x, where some
intervention needed if the primary name node went down.


I hope my math makes sense

On May 26, 2017 9:04 AM, "Susheel Kumar" <susheel2777@gmail.com> wrote:

>From ZK documentation, observers do not participate in vote,  so Pushkar,
when you said 5 nodes participate in voting, what exactly you mean?

-- Observers are non-voting members of an ensemble which only hear the
results of votes, not the agreement protocol that leads up to them.

Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
experiment is practically possible, correct?


On Fri, May 26, 2017 at 3:53 AM, Rick Leir <rleir@leirtech.com> wrote:

> Jan, Shawn, Susheel
>
> First steps first. First, let's do a fault-tolerant cluster, then maybe a
> _geographically_ fault-tolerant cluster.
>
> Add another server in either DC1 or DC2, in a separate rack, with
> independent power etc. As Shawn says below, install the third ZK there.
You
> would satisfy most of your requirements that way.
>
> cheers -- Rick
>
>
> On 2017-05-23 12:56 PM, Shawn Heisey wrote:
>
>> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
>>
>>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
>>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
>>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
dc2.
>>> (I didn't have the availability of 3rd data center for ZK so went with
only
>>> 2 data center with above configuration) and so far no issues. Its been
>>> running fine, indexing, replicating data, serving queries etc. So in my
>>> test, setting up single cluster across two zones/data center works
without
>>> any issue when there is no or very minimal latency (in my case around
30ms
>>> one way
>>>
>>
>> With that setup, if dc2 goes down, you're all good, but if dc1 goes down,
>> you're not.
>>
>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
>> unreachable, and SolrCloud is going to go read-only.  Queries would most
>> likely work, but you would not be able to change the indexes at all.
>>
>> ZooKeeper with N total servers requires int((N/2)+1) servers to be
>> operational to maintain quorum.  This means that with five total servers,
>> three must be operational and able to talk to each other, or ZK cannot
>> guarantee that there is no split-brain, so quorum is lost.
>>
>> ZK in two data centers will never be fully fault-tolerant. There is no
>> combination of servers that will work properly.  You must have three data
>> centers for a geographically fault-tolerant cluster.  Solr would be
>> optional in the third data center.  ZK must be installed in all three.
>>
>> Thanks,
>> Shawn
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message