cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nik Martin <>
Subject Re: Advanced Networking
Date Tue, 16 Oct 2012 20:39:29 GMT

I can tell you how I did full end-to-end redundancy that works and works 
well for me:

two L3 switches, stacked, RSTP and Broadcast Flood cpontrol off.

Split switch into three VLANS: Management, Guest+pub, and Storage.

ALL hypervisors are xenserver 6.02, and eac has 6 1GbE ports. I 
determined which ports were connected to which transcievers,and spread 
out the network such that no bonded interface shares an ethernet 
transciever, and from the back of the HV it looks like:

mgmt:pub :pub :strg:strg:mgmt
sw1 :sw1 :sw2 :sw1 :sw2 :sw2

So, I rounded up a windows laptop and installed xencenter, and bonded:
eth0 & eth5 became "management" tagged network in Xencenter )and 
Cloudstack). This bond is active/passive

eth1 & 2 were bonded to form "guest_pub" network in Xencenter and 
Cloudstack. This bond is active-active

eth3 & 4 were bonded to form "storage" network in Xencenter and 
Cloudstack. Also active-active, and 9000 MTU

My SAN node1 has a management port and a 10G uplink to one switch, and 
SAN node2(active-active SAN) has a 10G port uplink to the other switch.

Using this configuration, NO switch changes have to be made to 
accommodate the bonded interfaces, because the management interface is 
active-passive, and the others are active-active using mode6 I think, 
which is a WLB algorithm, which requires no switch changes whatsoever.

Additionally, the storage VLAN is setup with jumbo frames end-to-end, 
and has ALL broadcast storm control turned off.

I monitored the network for several days and have not noticed any floods 
or broadcast storms at all.

Hope this helps.

On 10/16/2012 03:00 PM, Trevor Francis wrote:
> So, with the help of the fine gents on the IRC channel (thank you KDamage), I have successfully
managed to crash my network over 20 times due to ARP floods coming from the bridges that I
have created. I attempted to create a highly available network by utilizing bonded bridges
split amongst trunked switch pairs and designing physical networks to take advantage of those
> Here is what my network looks like
> Host (6 total ethernet ports)
> Eth0, Eth1 = Bond0, cloudVirBr10 (bridge) . Bonding mode 6, STP enabled.
> Eth2, Eth3 = Bond2, cloudVirBr12 (bridge) . Bonding mode 6, STP enabled.
> Eth4, Eth5 = Bond1, cloudVirBr11 (bridge) . Bonding mode 6, STP enabled.
> I have the bonds split amongst trunked switches.
> Switch1A - eth0 , Switch1B, eth1 trunked between each other, STP enabled.
> Switch2A - eth4, Switch2B, eth5 trunked between each other, STP enabled.
> Switch3A - eth2, Switch3B, eth3 trunked between each other, STP enabled.
> The switch pairs are totally isolated from each other (ie, no interswitch trunking).
> So, you can turn off any switch or pull any cable and the network still works fine.
> Switch1A and Switch1B have internet delivered to them through HSRP (2 drops), so if either
of the switches crashes, the internet continues.
> I have been working for over a month to figure out why the floods occur, but we just
cant seem to figure it out.
> The hosts are running Cent 6.3 and KVM HVs.
> Our goal was to have Management and Storage traffic go over Bond2, Guest traffic go over
Bond1, and Public Internet traffic go over Bond0. We created physical networks in advanced
networking with those settings and KVM tags corresponding to our bridge names.
> Our application requirement require that we have 2 ethernet connections assigned directly
to the linux VMs (eth0 = public internet, eth1 = guest network). We cannot use NAT between
a public and a private network.
> To achieve this, we created a standard Isolated guest network and then created a shared
network that assigned a pool of public IPs directly to the machines.
> We can successfully turn a VM up with a public IP without storms and turn a VM up with
a guest IP without storms. But when we turn a VM up with both, the storms start immediately,
killing the public network.
> We are considering abandoing KVM in lieu of Xenserver, but we were told that there was
no way to successfully separate our public traffic to go over a bonded bride and have our
guest traffic go over a different bridge.
> My brain still works in "Old Networking" mode, so the new cloud networking stuff is throwing
me for a loop.
> Seeing what our network consists of, do you recommend doing things a different way? We
are considering swapping Xen for the KVM HVs and are looking at getting maximum performance
while still maintaining full 2N redundancy.
> Thanks,



Nik Martin
VP Business Development
Nfina Technologies, Inc.
+ x1003
Relentless Reliability

View raw message