Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: <CAM3razj_obbw0xej_iry7Q5Ay+LcMNB1SrfvYhXtDXCYviiFsw@mail.gmail.com>
References: <CAM3razgOhnwMB=EceiT=zrYf5-8DbEgYWYF29UL6kv49FfAmBg@mail.gmail.com>
 <0040A58A-7EC4-4FDD-BEC7-1A5C79BBAEFE@jordanzimmerman.com>
 <34C37303-1075-4DD5-8E83-248D2F5AE3F8@slac.stanford.edu> <CAM3razj_obbw0xej_iry7Q5Ay+LcMNB1SrfvYhXtDXCYviiFsw@mail.gmail.com>
From: Michael Han <hanm@cloudera.com>
Date: Mon, 6 Mar 2017 17:05:43 -0800
Message-ID: <CA+i0x1JXaexBYy4wu3ACuGy38UfO1TzgQpS_JY8zC+iJByKX_A@mail.gmail.com>
Subject: Re: Zookeeper Cross Datacenter Cluster
To: UserZooKeeper <user@zookeeper.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c191da04526be054a199e26
archived-at: Tue, 07 Mar 2017 01:06:28 -0000

--94eb2c191da04526be054a199e26
Content-Type: text/plain; charset=UTF-8

Back up requires replication which has two types, synchronous and
asynchronous.

ZooKeeper quorum provides synchronous replication. But as mentioned, 2 DC
will not work no matter how. You need at least three (and in general odd
numbers - for majority quorum). There are quorum weights and groups that
you might leverage but those would not solve 2 DC fault tolerant case.
There were a good discussion awhile ago about these concepts and multi-DC
set up for reference. [1]

ZooKeeper observer provides asynchronous replication. One setup is to have
a quorum of servers in one DC and having all observers in another. As with
any async replication, you might lose data, and whether or not this is OK
depends on your use case and your requirement on data integrity.

For fail over, using quorum you get it for free, as no extra work is
required. Using observer to fail over requires to reconfigure and restart
the ensemble by promoting observers to followers, so it is not
automatically provided by ZK and you'd need application logic to handle it
or do it manually.

There are other "external" approaches of backing up for example you can run
a background job to periodically backup ZK transaction log and snapshots
and store those somewhere then later spin a new ensemble and recover from
those. This approach is likely to lose data as well as its async
replication and it also requires application logic to handle the recover
process or do it manually...

[1]
http://zookeeper-user.578899.n2.nabble.com/zookeeper-deployment-strategy-for-multi-data-centers-td7582358.html

On Mon, Mar 6, 2017 at 2:42 PM, Le Cyberian <lecyberian@gmail.com> wrote:

> Thank for your replies.
>
> I see, in that way then failover won't be possible as if the main/major DC
> is down.
>
> Is there something i can do using hierarchical quorum groups or observers ?
>
> I am quiet confused to achieve this setup somehow, as how someone would
> build a failover over 2 dc's :-/
>
> On Mon, Mar 6, 2017 at 11:08 PM, Van Klaveren, Brian N. <
> bvan@slac.stanford.edu> wrote:
>
> > Or, an extra server in a third datacenter to break the tie.
> >
> > Brian
> >
> >
> > > On Mar 6, 2017, at 2:05 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com>
> > wrote:
> > >
> > > This scenario is not possible. An even number of servers doesn't help
> > you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each
> > DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the
> only
> > option is to choose one of the DCs as the main DC and put an extra server
> > there. This way you could lose the "minor" DC.
> > >
> > > -Jordan
> > >
> > >> On Mar 6, 2017, at 11:34 AM, Le Cyberian <lecyberian@gmail.com>
> wrote:
> > >>
> > >> Hi Guys,
> > >>
> > >> I would like to have Kafka Cluster which is dependent on Zookeeper
> like
> > >> many other great projects out their. Kafka would have 4 nodes as its
> > >> cluster since their having even number is not a problem because it
> does
> > >> leader election based its config it keeps in zookeeper.
> > >>
> > >> My scenario is having two server rooms on different floors having
> fiber
> > >> connectivity with each other so network latency / lag is almost like a
> > >> local connection.
> > >>
> > >> However, I would like to implement zookeeper cluster between both of
> > them
> > >> for auto-failover / redundancy purposes. So if one server room is down
> > >> other should work properly.
> > >>
> > >> For example: on a 5 node zookeeper cluster distributed on both of the
> > >> server rooms with 3 in one and 2 in other would give me 2 node failure
> > >> tolerance, However if the server room which has 3 servers is down then
> > >> ensemble would be broken and split brain.
> > >>
> > >> Can you please suggest how to achieve this to have failover /
> redundancy
> > >> between two server rooms or two locations lets say ?
> > >>
> > >> Is it possible to run 3 node cluster in each server room and have some
> > sort
> > >> of master-master between them ?
> > >>
> > >> Thank for your time and helping out in advance.
> > >>
> > >> Kind regards,
> > >>
> > >> Le
> > >
> >
> >
>


-- 
Cheers
Michael.

--94eb2c191da04526be054a199e26--