hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Managing multi-site clusters with Zookeeper
Date Mon, 15 Mar 2010 19:12:06 GMT

Michael Bauland wrote:
> - When I connect with a client to the Zookeeper ensemble I provide the
> three IP addresses of my three Zookeeper servers. Does the client then
> choose one of them arbitrarily or will it always try to connect to the
> first one first? I'm asking since I would like to have my clients first
> try to connect to the local Zookeeper server and only if that fails (for
> whatever reason, maybe it's down) it should try to connect to one of the
> servers on a different physical location.

It explicitly randomizes the list, this is to eliminate all clients 
connecting to the same servers in the same order, rather it distributes 
the load.

There's an option to turn this off in the c client, but not in the java 
client. However this is intended for testing purposes, not for 
production use. We could add it to the java client (create a JIRA if you 
like) however I'm not sure it would solve your problem. Once the local 
server is accessible again there'd be nothing to cause the client to 
connect to the local server. If the issue is intermittent it could 
non-optimal.

An option is to have more than one local server. So if you distrib btw 3 
colos then use 7 servers (3-2-2) and have the client connect to the 2 
local. This handles local failure, however it does not handle 
partitioning of the local servers from the remote ensemble members. 
However if both local servers are unable to connect to remote servers it 
seems likely that the client couldn't either (network partition). Not an 
optimal solution either unfortunately.

We have talked about adding this functionality to the ZK client (connect 
to the server with lowest latency/load/connections/etc... first). 
However this is not currently implemented. There's also the issue of how 
to decide when to switch to another server (ie when local comes back). 
For the time being you may have to handle this within your own code (two 
possible sessions based on connect string).

> 
>> You'll want to increase from the defaults since those are typically for
>> high performance interconnect (ie within colo). You are correct though,
>> much will depend on your env. and some tuning will be involved.
> 
> Do you have any suggestions for the parameters? So far I left tickTime
> at 2 sec and increased initLimit and syncLimit to 30 (i.e., one minute).
> 
> Our sites are connected with 1Gbit to the Internet, but of course we
> have no influence on what's in between. The data managed by zookeeper is
> quite large (snapshots are 700 MByte, but they may increase in the future).

What's you latency look like? Try using something like ping btw your 
colos over a longish period of time. Then look at the min/max/avg 
results that you see. What are you seeing? That along with bandwidth 
measurement (copy files using scp say) will help you decide.

We'd definitely be interested in your feedback as part of this process. 
Both in terms of docs (ff to enter jiras) and other insights you have as 
part of the effort.

Regards,

Patrick

Mime
View raw message