hadoop-zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Wright <wrig...@gmail.com>
Subject Re: Dynamic adding/removing ZK servers on client
Date Mon, 03 May 2010 17:28:29 GMT
> Could you provide some insight into why you need this? Just so we have addl
> background, I'm interested to know the use case.

Sure, we're building a clustered application that will use zookeeper
as part of it. We need to manage ZK ourself. The cluster running the
app & ZK may change over time (nodes added or removed) and we need to
keep ZK itself in-sync with any changes. They won't be common, but we
can't shut the app down to make the changes, it needs to be
transparent.

>
> Are you expecting all of the servers to change each time, or just
> incremental changes (add/remove a single server, vs say move the entire
> cluster from 3 hosts a/b/c to x/y/z)

I'd expect a small number of changes at any time - a few nodes being
added, a few nodes being removed. Most of the nodes will stay the
same.

>
> Any chance you could use DNS for this? ie change the mapping for the
> hostname from a -> x ip? Since the server a will go down anyway, this would
> cause the client to reconnect to b/c (eventually when dns ttl expires the
> client would also potentially connect to x).
> https://issues.apache.org/jira/browse/ZOOKEEPER-328
> https://issues.apache.org/jira/browse/ZOOKEEPER-338


Well, there are a lot of issues with DNS (including security & cache)
so I'd prefer to avoid it. Also, the real issue is the # of servers
are changing, not just their IP.
Although we probably wouldn't use it, I do think it would be nice to
support a single hostname for the ZK cluster with one A records for
each member, and have the ZK client handle resolving that properly
each time it connects.


> You might also look at this patch, we never committed it but it might be
> interesting to you:
> https://issues.apache.org/jira/browse/ZOOKEEPER-146
>
> The benefit is that you'd only have one place to make the change, esp given
> that clients might be down/unreachable when this change occurs. Clients
> would have to poll this service whenever they get disconnected from the
> ensemble. One drawback of this approach is that the HTTP now becomes a
> potential SPOF. (although I guess you could always fall back to something,
> or potentially have a list of HTTP hosts to do the lookup, etc...).

Well, that just handles distribution of the list (which isn't really
our problem), it doesn't help with restarting the ZK client when the
list changes - it only pulls the list once, so you still have to
completely shutdown and restart the ZK client.


> It does sound interesting, however once we add something like this it's hard
> to change given that we try very hard to maintain b/w compatibility. If you
> did the testing and were able to verify I don't see why we couldn't add it -
> as it's "optional" in the sense that it would only be called in the use case
> you describe. I would feel more confident if we had more concrete detail on
> how we intend to do 107 (a basic functional/design doc that at least reviews
> all the issues), and how this would fit in. But I don't see that should
> necessarily be a blocker (although others might feel differently).

Have you ever considered adding features like this via a protected
interface (i.e. the are useful but aren't fully standardized, so if a
client wants to use it they can sub-class ZK and make them public)?

The ability to dynamically modify the server list on the client side
seems like it would be required no matter what approach were taken to
dynamic clusters.

-Dave Wright

Mime
View raw message