lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: Spread SolrCloud across two locations
Date Fri, 26 May 2017 14:56:23 GMT
Thanks, Pushkar, Make sense.  Trying to understand the difference between
your setup vs Jan's proposed setup.

- Seems like when DC1 goes down, in your setup we have to bounce *one* from
observer to non-observer while in Jan's setup *two* observers to
non-observers.  Anything else I am missing

- When DC1 comes back -  with your setup we need to bounce the one
non-observer to observer to have 5 nodes quorum otherwise there are 3 + 3
observers while with Jan's setup if we don't take any action when DC1 comes
back, we are still operational with 5 nodes quorum.  Isn't it?  Or I am
missing something.



On Fri, May 26, 2017 at 10:07 AM, Pushkar Raste <pushkar.raste@gmail.com>
wrote:

> Damn,
> Math is hard
>
> DC1 : 3 non observers
> DC2 : 2 non observers
>
> 3 + 2 = 5 non observers
>
> Observers don't participate in voting = non observers participate in voting
>
> 5 non observers = 5 votes
>
> In addition to the 2 non observer, DC2 also has an observer, which as you
> pointed out does not participate in the voting.
>
> We still have 5 voting nodes.
>
>
> Think of the observer as a standby name node in Hadoop 1.x, where some
> intervention needed if the primary name node went down.
>
>
> I hope my math makes sense
>
> On May 26, 2017 9:04 AM, "Susheel Kumar" <susheel2777@gmail.com> wrote:
>
> From ZK documentation, observers do not participate in vote,  so Pushkar,
> when you said 5 nodes participate in voting, what exactly you mean?
>
> -- Observers are non-voting members of an ensemble which only hear the
> results of votes, not the agreement protocol that leads up to them.
>
> Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
> experiment is practically possible, correct?
>
>
> On Fri, May 26, 2017 at 3:53 AM, Rick Leir <rleir@leirtech.com> wrote:
>
> > Jan, Shawn, Susheel
> >
> > First steps first. First, let's do a fault-tolerant cluster, then maybe a
> > _geographically_ fault-tolerant cluster.
> >
> > Add another server in either DC1 or DC2, in a separate rack, with
> > independent power etc. As Shawn says below, install the third ZK there.
> You
> > would satisfy most of your requirements that way.
> >
> > cheers -- Rick
> >
> >
> > On 2017-05-23 12:56 PM, Shawn Heisey wrote:
> >
> >> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> >>
> >>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
> in
> >>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
> >>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
> dc2.
> >>> (I didn't have the availability of 3rd data center for ZK so went with
> only
> >>> 2 data center with above configuration) and so far no issues. Its been
> >>> running fine, indexing, replicating data, serving queries etc. So in my
> >>> test, setting up single cluster across two zones/data center works
> without
> >>> any issue when there is no or very minimal latency (in my case around
> 30ms
> >>> one way
> >>>
> >>
> >> With that setup, if dc2 goes down, you're all good, but if dc1 goes
> down,
> >> you're not.
> >>
> >> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> >> unreachable, and SolrCloud is going to go read-only.  Queries would most
> >> likely work, but you would not be able to change the indexes at all.
> >>
> >> ZooKeeper with N total servers requires int((N/2)+1) servers to be
> >> operational to maintain quorum.  This means that with five total
> servers,
> >> three must be operational and able to talk to each other, or ZK cannot
> >> guarantee that there is no split-brain, so quorum is lost.
> >>
> >> ZK in two data centers will never be fully fault-tolerant. There is no
> >> combination of servers that will work properly.  You must have three
> data
> >> centers for a geographically fault-tolerant cluster.  Solr would be
> >> optional in the third data center.  ZK must be installed in all three.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message