hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Re: Multiple ZK clusters or a single, shared cluster?
Date Fri, 17 Jul 2009 20:38:09 GMT
we designed zk to have high performance so that it can be shared by 
multiple applications. the main thing is that you use dedicated zk 
machines (with a dedicated disk for logging). once you have that in 
place, watch the load on your cluster, as long as you aren't saturating 
the cluster you should share.

as you point out running multiple clusters is a hardware investment, 
plus you miss out on opportunities to improve reliability. for example, 
if you have three applications that have a cluster of 3 zk servers each, 
one failure will result in an outage. if instead of using the 9 servers 
you have the same three applications use a zk cluster with 7 servers you 
can tolerate three failures without an outage.

the key of course is to make sure that you don't oversubscribe the server.

ben

Jonathan Gray wrote:
> Hey guys,
>
> Been using ZK indirectly for a few months now in the HBase and Katta 
> realms.  Both of these applications make it really easy so you don't 
> have to be involved much with managing your ZK cluster to support it.
>
> I'm now using ZK for a bunch of things internally, so now I'm manually 
> configuring, starting, and managing a cluster.
>
> What advice is there about whether I should be sharing a single cluster 
> between all my applications, or running separate ones for each use?
>
> I've been told that it's strongly recommended to run your ZK nodes 
> separately from the application using them (this is actually what we're 
> telling new users over in HBase, though a majority of installations will 
> likely co-host them with DataNodes and RegionServers).
>
> I don't have the resources to maintain a separate 3+ node ZK cluster for 
> each of my applications, so this is not really an option.  I'm trying to 
> decide if I should have HBase running/managing it's own ZK cluster that 
> is co-located with some of the regionservers (there will be ample 
> memory, but ZK will not have a dedicated disk), or if I should be 
> pointing it to a dedicated 3 node ZK cluster.
>
> I would then also have Katta pointing at this same shared cluster (or a 
> separate cluster would be co-located with katta nodes).  Same for my 
> application; could share nodes with the app servers or pointed at a 
> single ZK cluster.
>
> Trade-offs I should be aware of?  Current best practices?
>
> Any help would be much appreciated.  Thanks.
>
> Jonathan Gray
>   


Mime
View raw message