cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Angus <>
Subject RE: Multi-Datacenter Deployment
Date Wed, 07 Jan 2015 10:12:11 GMT
Hi Logan,

The biggest issue to deal with in a multi-datacenter deployment is the database, specifically
latency between database nodes and between the CloudStack management servers and the database.
 Hosts in remote zones quite happily talk back to remote management servers over WAN links.

Currently CloudStack employs record locking which is not multi-master writing friendly. So
you can only really write to one node. (this is something that is being looked at).

As the latency between the management servers and the hosts is far less important than that
relating to DB traffic the standard configuration is to have a master zone which contains
your primary management server(s) and a MySQL master and slave pair. You then have a second
zone (chosen largely by the latency on the data link) which contains secondary management
server(s) and MySQL slave to the primary zone MySQL master (and probably a slave to that slave
- I'll call it a secondary slave).  Then you would you global server load balancing (GSLB)
to switch from the primary zone to the secondary zone in the event of a primary zone failure.
 The slave in the secondary zone becomes the master and the secondary slave becomes 'the'

All other zones use the primary zone mgmt. infrastructure unless GSLB directs them to the
secondary zone.


Paul Angus
Cloud Architect
S: +44 20 3603 0540 | M: +447711418784 | T: @CloudyAngus

-----Original Message-----
From: Logan Barfield []
Sent: 06 January 2015 19:46
Subject: Multi-Datacenter Deployment

We are currently running a single location CloudStack deployment:
- 1 Hardware firewall
- 1 Mangement/Database Server
- 1 NFS staging store (for S3 secondary storage)
- Ceph RBD for primary storage
- 4 Hypervisors
- 1 Zone/Pod/Cluster

We are looking to expand our deployment to other datacenters, and I'm trying to determine
the best way to go about it.  The documentation is a bit lacking for multi-site deployments.

Our goal for the multi-site deployment is to have a zone for each site (E.G. US East, US West,
Europe) that our customers can use to deploy instances in their preferred geographic area.

Since we don't want to have different accounts for every datacenter, I don't think using Regions
makes sense for us (and I'm not sure what they're actually good for without keeping accounts/users/domains
in sync).

Right now I'm thinking our setup will be as follows:
- Firewall, Management Server, NFS staging server, primary storage, and Hypervisors in each
- All Management servers will be on the same management network.
- Management servers will be connected via site-to-site VPN links over WAN.
- MySQL replication (Percona?) will be set up on the management servers.
Having an odd number of servers to protect against split brain, and keeping redundant database
- One region (default)
- One zone for each datacenter
- Geo-enabled DNS to direct customers to the nearest Management server
- Object storage for secondary storage across cloud.

My primary concerns with this setup are:
- I haven't really seen multi-site deployments details anywhere.
- Potential for split-brain.
- How will HA be handled (e.g., if a VPN link goes down and one of the remote management servers
can't contact a host, will it try to initiate
HA?) - This sort of goes along with the split brain problem.

Are my assumptions here sound, or is there a standard recommended way of doing multi-site

Any suggestions are much appreciated.
Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<>
CSForge – rapid IaaS deployment framework<>
CloudStack Consulting<>
CloudStack Software Engineering<>
CloudStack Infrastructure Support<>
CloudStack Bootcamp Training Courses<>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
View raw message