cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kelceydamage@bbits" <>
Subject Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
Date Fri, 17 Aug 2012 15:46:36 GMT
Read that IBM article you referenced, it made a lot of things clear to me.

Sent from my iPhone

On Aug 17, 2012, at 5:54 AM, "Matthew Patton" <> wrote:

> 'cloudVirBr' is hard-coded in /usr/lib[64]/cloud/agent/scripts/vm/network/vnet/
in function addVlan().
> What I don't understand is how the script is even working. It does
> vconfig add ethN V
> brctl addbr brV
> brctl addif brV ethN.V
> Furthermore, in scripts/storage/qcow2/ (anyone looked at why there are 3
different copies of this shell script?!?) it does the same thing but using bonds.
> But when I run thru the sequence by hand (eth or bond) it doesn't pass traffic. The only
way I get it to work is:
> brctl addbr br
> brctl addif br ethN
> vconfig add ethN V
> ifconfig ethN.V <ipaddr> <netmask>
> And I can define as many ethN.[1..9] as I like with all different VLAN IDs and attach
them to the single bridge. My way also works when adding a bond to the bridge and adding as
many VLAN tagged bonds as I like.
> But the guest isn't receiving response packets to things like ARP. When I sniff bond0
on the hypervisor I see the VLAN tag in the frame AS LONG AS in the guest I configure ethN.V;
as would be logical. The ARP reply also gets as far as bond0 on the way back. Using tcpdump
on interface 'vnet0' doesn't show any packets going in either direction though they are transiting
bond0 just fine. I can see the ARP replies on bond0.N which is again as expected.
> If I can sniff bond0, how come I can't sniff vnet0?
> There is something odd in the 'brctl showmacs <bridge>' output which probably explains
why vnet0 sees no packets.
> The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and eventually
ages out. But there is another MAC on port 2 that is the same except it starts with 'fe'.
This one is marked local=yes and doesn't age out. It seems this is the MAC given to the host-side
of the tap. So I can understand it being semi-permanent. So then why is the the guest's (?private?)
MAC address showing up in the bridge? If the ARP response packets have the wrong (guest's
private MAC) in them (and they do) that would explain why it can't return to the guest since
it "missed" the on-ramp that is the hypervisor's end of the TUN interface.
> Looks like a mistake in the tap code that's forgetting to rewrite the MAC value? Maybe
because there is a VLAN tag in the frame?
> At first there were no "QEMU networks" defined since I had undefined the default one
since I wasn't interested in NAT.
> [root@kvm1b ~]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> But seeing as that might be a problem I defined one for bridging.
> <network>
>  <name>bridge-shared</name>
>  <uuid>0824e08b-bb0e-452a-e2c9-dcca52af2341</uuid>
>  <forward mode='bridge'/>
>  <bridge name='shared' />
> </network>
> Now I have a network
> [root@kvm1b networks]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> bridge-shared        active     yes
> This is the interface section from my guest's XML.
>    <interface type='bridge'>
>      <mac address='52:54:00:c9:8d:d1'/>
>      <source bridge='shared'/>
>      <model type='virtio'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>    </interface>
> [root@kvm1b ~]# virsh domiflist test
> Interface  Type       Source     Model       MAC
> -------------------------------------------------------
> vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1
> [root@kvm1b networks]# virsh iface-list
> Name                 State      MAC Address
> --------------------------------------------
> bond0.39             active     00:25:90:4c:ba:92
> bond0.398            active     00:25:90:4c:ba:92
> eth2                 active     90:e2:ba:04:f6:28
> eth3                 active     90:e2:ba:04:f6:29
> lo                   active     00:00:00:00:00:00
> shared               active     00:25:90:4c:ba:92
> and for good measure
> [root@kvm1b networks]# virsh iface-dumpxml shared
> <interface type='bridge' name='shared'>
>  <protocol family='ipv6'>
>    <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
>  </protocol>
>  <bridge>
>    <interface type='bond' name='bond0'>
>      <bond>
>        <interface type='ethernet' name='eth0'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>        <interface type='ethernet' name='eth1'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>      </bond>
>    </interface>
>    <interface type='ethernet' name='vnet0'>
>      <mac address='fe:54:00:c9:8d:d1'/>
>    </interface>
>  </bridge>
> </interface>
> [root@kvm1b networks]# virsh domifstat test vnet0
> vnet0 rx_bytes 44592
> vnet0 rx_packets 847
> vnet0 rx_errs 0
> vnet0 rx_drop 0
> vnet0 tx_bytes 2150
> vnet0 tx_packets 35
> vnet0 tx_errs 0
> vnet0 tx_drop 0
> So you'd think something was getting back to the guest. I just don't know what. tcpdump
running inside the guest on eth0 shows nothing at all, just like sniffing vnet0.
> The MAC hitting the wire on the hypervisor's bond0 is still the guest's private MAC and
it seems the response packets don't know where to go once they come back to the bridge.
> On a different note I ran across this:
> So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!

View raw message