Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.171 is neither permitted
 nor denied by domain of phunt@apache.org)
Message-ID: <4BDEFB1A.5010107@apache.org>
Date: Mon, 03 May 2010 09:34:34 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US;
 rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: zookeeper-user@hadoop.apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>
CC: Dave Wright <wrightd@gmail.com>
Subject: Re: Dynamic adding/removing ZK servers on client
References: <p2sf16bc8301005030703j5a0264e2s2e9ff2cf1339bd2b@mail.gmail.com>
In-Reply-To: <p2sf16bc8301005030703j5a0264e2s2e9ff2cf1339bd2b@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


On 05/03/2010 07:03 AM, Dave Wright wrote:
> I've got a situation where I essentially need dynamic cluster
> membership, which has been talked about in ZOOKEEPER-107 but doesn't
> look like it's going to happen any time soon.
>

Could you provide some insight into why you need this? Just so we have 
addl background, I'm interested to know the use case.

> For now, I'm planning on working around this by having a simple
> coordinator service on the server nodes that will re-write the configs
> and bounce the servers when membership changes. Clients will may get
> an error or two and need to reconnect, but that should be handled by
> the normal error logic.
>

Are you expecting all of the servers to change each time, or just 
incremental changes (add/remove a single server, vs say move the entire 
cluster from 3 hosts a/b/c to x/y/z)

> On the client side, I'd really like to dynamically update the server
> list w/o having to re-create the entire Zookeeper object. Looking at
> the code, it seems like it would be pretty trivial to add
> "RemoveServer()/AddServer()" functions for Zookeeper that calls down
> to ClientCnxn, where they are just maintained in a list. Of course if
> the server being removed is the one currently connected, we'd need to
> disconnect, but a simple call to disconnect() seems like it would
> resolve that and trigger the automatic re-connection logic.
>

You would hook this (add/remove) into JMX? That seems like a good option 
to provide.

Any chance you could use DNS for this? ie change the mapping for the 
hostname from a -> x ip? Since the server a will go down anyway, this 
would cause the client to reconnect to b/c (eventually when dns ttl 
expires the client would also potentially connect to x).

If this is an option be sure to see (a bit of work to do):
https://issues.apache.org/jira/browse/ZOOKEEPER-328
https://issues.apache.org/jira/browse/ZOOKEEPER-338

You might also look at this patch, we never committed it but it might be 
interesting to you:
https://issues.apache.org/jira/browse/ZOOKEEPER-146

The benefit is that you'd only have one place to make the change, esp 
given that clients might be down/unreachable when this change occurs. 
Clients would have to poll this service whenever they get disconnected 
from the ensemble. One drawback of this approach is that the HTTP now 
becomes a potential SPOF. (although I guess you could always fall back 
to something, or potentially have a list of HTTP hosts to do the lookup, 
etc...).

> Does anyone see an issue with that approach?
> Were I to create the patch, do you think it would be interesting
> enough to merge? It seems like that functionality will eventually be
> needed for whatever full dynamic server support is eventually
> implemented.

It does sound interesting, however once we add something like this it's 
hard to change given that we try very hard to maintain b/w 
compatibility. If you did the testing and were able to verify I don't 
see why we couldn't add it - as it's "optional" in the sense that it 
would only be called in the use case you describe. I would feel more 
confident if we had more concrete detail on how we intend to do 107 (a 
basic functional/design doc that at least reviews all the issues), and 
how this would fit in. But I don't see that should necessarily be a 
blocker (although others might feel differently).

(fyi it's good to discuss this sort of thing on zookeeper-dev, please 
move responses to that list)

Sounds like an useful project, I'm interested to her what others think 
about it. Regards,

Patrick