cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <>
Subject Re: VXLAN and KVm experiences
Date Tue, 23 Oct 2018 09:21:34 GMT
Hi Wido,

I have "pioneered" this one in production for last 3 years (and suffered a
nasty pain of silent drop of packages on kernel 3.X back in the days
because of being unaware of max_igmp_memberships kernel parameters, so I
have updated the manual long time ago).

I never had any issues (beside above nasty one...) and it works very well.
To avoid above issue that I described - you should increase
max_igmp_memberships (/proc/sys/net/ipv4/igmp_max_memberships)  - otherwise
with more than 20 vxlan interfaces, some of them will stay in down state
and have a hard traffic drop (with proper message in agent.log) with kernel
>4.0 (or I silent, bitchy random packet drop on kernel 3.X...) - and also
pay attention to MTU size as well - anyway everything is in the manual (I
updated everything I though was missing) - so please check it.

Our example setup:

We have i.e. bond.950 as the main VLAN which will carry all vxlan "tunnels"
- so this is defined as KVM traffic label. In our case it didn't make sense
to use bridge on top of this bond0.950 (as the traffic label) - you can
test it on your own - since this bridge is used only to extract child
bond0.950 interface name, then based on vxlan ID, ACS will provision and join this new vxlan interface to NEW bridge created
(and then of course vNIC goes to this new bridge), so original bridge (to
which belonged) is not used for anything.

Here is sample from above for vxlan 867 used for tenant isolation:

root@hostname:~# brctl show brvx-867

bridge name     bridge id               STP enabled     interfaces
brvx-867                8000.2215cfce99ce       no              vnet6


root@hostname:~# ip -d link show vxlan867

297: vxlan867: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8142 qdisc noqueue
master brvx-867 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 22:15:cf:ce:99:ce brd ff:ff:ff:ff:ff:ff promiscuity 1
    vxlan id 867 group dev bond0.950 port 0 0 ttl 10 ageing 300

root@ix1-c7-2:~# ifconfig bond0.950 | grep MTU

So note how the vxlan interface has by 50 bytes smaller MTU than the
bond0.950 parent interface (which could affects traffic inside VM) - so
jumbo frames are needed anyway on the parent interface (bond.950 in example
above with minimum of 1550 MTU)

Ping me if more details needed, happy to help.


On Tue, 23 Oct 2018 at 08:23, Wido den Hollander <> wrote:

> Hi,
> I just wanted to know if there are people out there using KVM with
> Advanced Networking and using VXLAN for different networks.
> Our main goal would be to spawn a VM and based on the network the NIC is
> in attach it to a different VXLAN bridge on the KVM host.
> It seems to me that this should work, but I just wanted to check and see
> if people have experience with it.
> Wido


Andrija Panić

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message