incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Patton" <>
Subject Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
Date Fri, 17 Aug 2012 12:54:33 GMT
'cloudVirBr' is hard-coded in  
/usr/lib[64]/cloud/agent/scripts/vm/network/vnet/ in function  

What I don't understand is how the script is even working. It does

vconfig add ethN V
brctl addbr brV
brctl addif brV ethN.V

Furthermore, in scripts/storage/qcow2/ (anyone looked at why  
there are 3 different copies of this shell script?!?) it does the same  
thing but using bonds.

But when I run thru the sequence by hand (eth or bond) it doesn't pass  
traffic. The only way I get it to work is:

brctl addbr br
brctl addif br ethN
vconfig add ethN V
ifconfig ethN.V <ipaddr> <netmask>

And I can define as many ethN.[1..9] as I like with all different VLAN IDs  
and attach them to the single bridge. My way also works when adding a bond  
to the bridge and adding as many VLAN tagged bonds as I like.

But the guest isn't receiving response packets to things like ARP. When I  
sniff bond0 on the hypervisor I see the VLAN tag in the frame AS LONG AS  
in the guest I configure ethN.V; as would be logical. The ARP reply also  
gets as far as bond0 on the way back. Using tcpdump on interface 'vnet0'  
doesn't show any packets going in either direction though they are  
transiting bond0 just fine. I can see the ARP replies on bond0.N which is  
again as expected.

If I can sniff bond0, how come I can't sniff vnet0?
There is something odd in the 'brctl showmacs <bridge>' output which  
probably explains why vnet0 sees no packets.

The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and  
eventually ages out. But there is another MAC on port 2 that is the same  
except it starts with 'fe'. This one is marked local=yes and doesn't age  
out. It seems this is the MAC given to the host-side of the tap. So I can  
understand it being semi-permanent. So then why is the the guest's  
(?private?) MAC address showing up in the bridge? If the ARP response  
packets have the wrong (guest's private MAC) in them (and they do) that  
would explain why it can't return to the guest since it "missed" the  
on-ramp that is the hypervisor's end of the TUN interface.

Looks like a mistake in the tap code that's forgetting to rewrite the MAC  
value? Maybe because there is a VLAN tag in the frame?

At first there were no "QEMU networks" defined since I had undefined the  
default one since I wasn't interested in NAT.
[root@kvm1b ~]# virsh net-list
Name                 State      Autostart

But seeing as that might be a problem I defined one for bridging.
   <forward mode='bridge'/>
   <bridge name='shared' />

Now I have a network
[root@kvm1b networks]# virsh net-list
Name                 State      Autostart
bridge-shared        active     yes

This is the interface section from my guest's XML.
     <interface type='bridge'>
       <mac address='52:54:00:c9:8d:d1'/>
       <source bridge='shared'/>
       <model type='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'  

[root@kvm1b ~]# virsh domiflist test
Interface  Type       Source     Model       MAC
vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1

[root@kvm1b networks]# virsh iface-list
Name                 State      MAC Address
bond0.39             active     00:25:90:4c:ba:92
bond0.398            active     00:25:90:4c:ba:92
eth2                 active     90:e2:ba:04:f6:28
eth3                 active     90:e2:ba:04:f6:29
lo                   active     00:00:00:00:00:00
shared               active     00:25:90:4c:ba:92

and for good measure

[root@kvm1b networks]# virsh iface-dumpxml shared
<interface type='bridge' name='shared'>
   <protocol family='ipv6'>
     <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
     <interface type='bond' name='bond0'>
         <interface type='ethernet' name='eth0'>
           <mac address='00:25:90:4c:ba:92'/>
         <interface type='ethernet' name='eth1'>
           <mac address='00:25:90:4c:ba:92'/>
     <interface type='ethernet' name='vnet0'>
       <mac address='fe:54:00:c9:8d:d1'/>

[root@kvm1b networks]# virsh domifstat test vnet0
vnet0 rx_bytes 44592
vnet0 rx_packets 847
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 2150
vnet0 tx_packets 35
vnet0 tx_errs 0
vnet0 tx_drop 0

So you'd think something was getting back to the guest. I just don't know  
what. tcpdump running inside the guest on eth0 shows nothing at all, just  
like sniffing vnet0.

The MAC hitting the wire on the hypervisor's bond0 is still the guest's  
private MAC and it seems the response packets don't know where to go once  
they come back to the bridge.

On a different note I ran across this:
So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!

View raw message