Return-Path: 
 <cloudstack-dev-return-4114-apmail-incubator-cloudstack-dev-archive=incubator.apache.org@incubator.apache.org>
X-Original-To: apmail-incubator-cloudstack-dev-archive@minotaur.apache.org
Delivered-To: apmail-incubator-cloudstack-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5FA6DADF
	for <apmail-incubator-cloudstack-dev-archive@minotaur.apache.org>;
 Fri, 17 Aug 2012 12:55:11 +0000 (UTC)
Received: (qmail 92618 invoked by uid 500); 17 Aug 2012 12:55:11 -0000
Delivered-To: apmail-incubator-cloudstack-dev-archive@incubator.apache.org
Received: (qmail 92523 invoked by uid 500); 17 Aug 2012 12:55:10 -0000
Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:cloudstack-dev-help@incubator.apache.org>
List-Unsubscribe: <mailto:cloudstack-dev-unsubscribe@incubator.apache.org>
List-Post: <mailto:cloudstack-dev@incubator.apache.org>
List-Id: <cloudstack-dev.incubator.apache.org>
Reply-To: cloudstack-dev@incubator.apache.org
Delivered-To: mailing list cloudstack-dev@incubator.apache.org
Received: (qmail 92513 invoked by uid 99); 17 Aug 2012 12:55:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Aug 2012 12:55:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FILL_THIS_FORM_SHORT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mpatton@inforelay.com
 designates 50.56.211.18 as permitted sender)
Received: from [50.56.211.18] (HELO mail2.inforelay.com) (50.56.211.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Aug 2012 12:55:01 +0000
Received: from omicron.sitelutions.com (omicron.sitelutions.com
 [67.208.74.181])
	by mail2.inforelay.com (Postfix) with ESMTP id B3412620DD
	for <cloudstack-dev@incubator.apache.org>;
 Fri, 17 Aug 2012 12:54:39 +0000 (UTC)
Received: from office-asa.iad4.inforelay.net ([69.169.88.196]
 helo=mpatton-pc.inforelay.net)
	by omicron.sitelutions.com with esmtp (Exim 4.69)
	(envelope-from <mpatton@inforelay.com>)
	id 1T2M4V-000478-3w
	for cloudstack-dev@incubator.apache.org; Fri, 17 Aug 2012 08:54:39 -0400
Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes
To: cloudstack-dev@incubator.apache.org
Subject: Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
References: 
 <CADRU-YB6vKcqHf_7BV2Q52pE-BJR=8rmEzPPfXufQi086mWi7A@mail.gmail.com>
 <021301cd7b11$88db0d10$9a912730$@bbits.ca>
 <CADRU-YA0ZV+kN4t1E-Jp9PFLbyXoqS88RVEbp+FtS1W1jmCcWQ@mail.gmail.com>
 <CADRU-YALbggBZwJLHtpeW3J1Q_pikEmoqbRsaStdJO9GCL1gng@mail.gmail.com>
 <D265D8B3-3453-428C-A6C1-3BD2AB33CBA5@bbits.ca>
Date: Fri, 17 Aug 2012 08:54:33 -0400
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: "Matthew Patton" <mpatton@inforelay.com>
Organization: InfoRelay
Message-ID: <op.wi6rg7izmtfvvt@mpatton-pc.inforelay.net>
In-Reply-To: <D265D8B3-3453-428C-A6C1-3BD2AB33CBA5@bbits.ca>
User-Agent: Opera Mail/11.62 (Win32)

'cloudVirBr' is hard-coded in  
/usr/lib[64]/cloud/agent/scripts/vm/network/vnet/modifyvlan.sh in function  
addVlan().

What I don't understand is how the script is even working. It does

vconfig add ethN V
brctl addbr brV
brctl addif brV ethN.V

Furthermore, in scripts/storage/qcow2/modifyvlan.sh (anyone looked at why  
there are 3 different copies of this shell script?!?) it does the same  
thing but using bonds.

But when I run thru the sequence by hand (eth or bond) it doesn't pass  
traffic. The only way I get it to work is:

brctl addbr br
brctl addif br ethN
vconfig add ethN V
ifconfig ethN.V <ipaddr> <netmask>

And I can define as many ethN.[1..9] as I like with all different VLAN IDs  
and attach them to the single bridge. My way also works when adding a bond  
to the bridge and adding as many VLAN tagged bonds as I like.

But the guest isn't receiving response packets to things like ARP. When I  
sniff bond0 on the hypervisor I see the VLAN tag in the frame AS LONG AS  
in the guest I configure ethN.V; as would be logical. The ARP reply also  
gets as far as bond0 on the way back. Using tcpdump on interface 'vnet0'  
doesn't show any packets going in either direction though they are  
transiting bond0 just fine. I can see the ARP replies on bond0.N which is  
again as expected.

If I can sniff bond0, how come I can't sniff vnet0?
There is something odd in the 'brctl showmacs <bridge>' output which  
probably explains why vnet0 sees no packets.

The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and  
eventually ages out. But there is another MAC on port 2 that is the same  
except it starts with 'fe'. This one is marked local=yes and doesn't age  
out. It seems this is the MAC given to the host-side of the tap. So I can  
understand it being semi-permanent. So then why is the the guest's  
(?private?) MAC address showing up in the bridge? If the ARP response  
packets have the wrong (guest's private MAC) in them (and they do) that  
would explain why it can't return to the guest since it "missed" the  
on-ramp that is the hypervisor's end of the TUN interface.


Looks like a mistake in the tap code that's forgetting to rewrite the MAC  
value? Maybe because there is a VLAN tag in the frame?


At first there were no "QEMU networks" defined since I had undefined the  
default one since I wasn't interested in NAT.
[root@kvm1b ~]# virsh net-list
Name                 State      Autostart
-----------------------------------------

But seeing as that might be a problem I defined one for bridging.
<network>
   <name>bridge-shared</name>
   <uuid>0824e08b-bb0e-452a-e2c9-dcca52af2341</uuid>
   <forward mode='bridge'/>
   <bridge name='shared' />
</network>


Now I have a network
[root@kvm1b networks]# virsh net-list
Name                 State      Autostart
-----------------------------------------
bridge-shared        active     yes


This is the interface section from my guest's XML.
     <interface type='bridge'>
       <mac address='52:54:00:c9:8d:d1'/>
       <source bridge='shared'/>
       <model type='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'  
function='0x0'/>
     </interface>

[root@kvm1b ~]# virsh domiflist test
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1

[root@kvm1b networks]# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
bond0.39             active     00:25:90:4c:ba:92
bond0.398            active     00:25:90:4c:ba:92
eth2                 active     90:e2:ba:04:f6:28
eth3                 active     90:e2:ba:04:f6:29
lo                   active     00:00:00:00:00:00
shared               active     00:25:90:4c:ba:92

and for good measure

[root@kvm1b networks]# virsh iface-dumpxml shared
<interface type='bridge' name='shared'>
   <protocol family='ipv6'>
     <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
   </protocol>
   <bridge>
     <interface type='bond' name='bond0'>
       <bond>
         <interface type='ethernet' name='eth0'>
           <mac address='00:25:90:4c:ba:92'/>
         </interface>
         <interface type='ethernet' name='eth1'>
           <mac address='00:25:90:4c:ba:92'/>
         </interface>
       </bond>
     </interface>
     <interface type='ethernet' name='vnet0'>
       <mac address='fe:54:00:c9:8d:d1'/>
     </interface>
   </bridge>
</interface>

[root@kvm1b networks]# virsh domifstat test vnet0
vnet0 rx_bytes 44592
vnet0 rx_packets 847
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 2150
vnet0 tx_packets 35
vnet0 tx_errs 0
vnet0 tx_drop 0

So you'd think something was getting back to the guest. I just don't know  
what. tcpdump running inside the guest on eth0 shows nothing at all, just  
like sniffing vnet0.

The MAC hitting the wire on the hypervisor's bond0 is still the guest's  
private MAC and it seems the response packets don't know where to go once  
they come back to the bridge.


On a different note I ran across this:  
http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbptap.htm
So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!