hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Greenwood" <to...@audiencescience.com>
Subject RE: Zookeeper WAN Configuration
Date Fri, 24 Jul 2009 23:23:30 GMT
Ted, could you elaborate a bit more on this? I was under the (mis)
impression that each ZK server in an ensemble only needed connectivity
to another member in the ensemble, not to each member in the ensemble.
It sounds like you are saying the latter is true.

Could you explain the idea behind the Observers feature, what this
concept is supposed to address, and how it applies to the WAN
configuration problem 
in particular?
"""
The ideas for federating ZK or allowing observers would likely do what
you
want.  I can imagine that an observer would only care that it can see
it's
local peers and one of the observers would be elected to get updates
(and
thus would care about the central service).
"""
This certainly sounds like exactly what I want...Was this introduced in
3.2 in full, or only partially?


Here, do you mean the servers will log warnings untill all the ensemble
members are visible to each other?
"""
Any servers that see a minority of the other servers will go tharn until
the
"partition" is healed.  That isn't what you want (at all).
"""

Given that 3.2 has a serious bug, I've recommended that we proceed with
our deploy based 3.1.1. For this version, it sounds like we will have to
open up connectivity from each zk server to each zk server, across the
various zones in the WAN. Is this correct?

-Todd

-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com] 
Sent: Friday, July 24, 2009 3:41 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Zookeeper WAN Configuration

Vanilla ZK servers will see security constraints as a network partition.
Any servers that see a minority of the other servers will go tharn until
the
"partition" is healed.  That isn't what you want (at all).

The ideas for federating ZK or allowing observers would likely do what
you
want.  I can imagine that an observer would only care that it can see
it's
local peers and one of the observers would be elected to get updates
(and
thus would care about the central service).

On Fri, Jul 24, 2009 at 3:32 PM, Todd Greenwood
<toddg@audiencescience.com>wrote:

> Like most folks, our WAN is composed of various zones, some central
> processing, some edge, some corp, and some in between (DMZs). In this
> model, a given Zookeeper server will not have direct connectivity to
all
> of it's peers in the ensemble due to various security constraints. Is
> this a problem? Are there special configurations for this model?
>
> Given 3 Zones
> -------------
>
> A <--> B
>         B <--> C
>
> A cannot see C, and vice versa.
> B can see A and C.
>
> 1. Will zookeeper servers function properly even if a given set of
> servers can only see some of the servers in the ensemble? For example,
> the shared config lists all zk servers in A, B, and C, but A can only
> see B, C can only see B, and B can see both A and C.
>
> 2. Will zookeeper servers flood the log with error messages if only a
> subset of the ensemble members are visible?
>
> 3. Will the zk ensemble function properly if the config used by each
> server only lists the servers in the ensemble that are visible?
Suppose
> that A has a config that only list servers in A and B, C a config for
C
> and B, and B has a config that lists servers in A, B, and C. Is this
the
> recommended approach?
>
> http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperAdmin.html
>



-- 
Ted Dunning, CTO
DeepDyve

Mime
View raw message