cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hugo Trippaers <HTrippa...@schubergphilis.com>
Subject RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS support in KVM
Date Thu, 25 Apr 2013 13:19:13 GMT
Heya,

Comments inline.

I ran through the entire procedure I use to test the setup and documented it as detailed as
possible. Hope this helps:

#
# CentOS KVM installation
#

Base installation: 
	CentOS 6.2 Basic Server 

Upgraded to CentOS 6.4:
	# yum update

Install Virtualization tools
	# yum groupinstall Virtualization "Virtualization Client" "Virtualization Platform" "Virtualization
Tools"

Verify Installed:
  hypervkvpd.x86_64 0:0-0.9.el6                    libguestfs.x86_64 1:1.16.34-2.el6     
        libvirt.x86_64 0:0.10.2-18.el6_4.3
  libvirt-client.x86_64 0:0.10.2-18.el6_4.3        python-virtinst.noarch 0:0.600.0-15.el6
       qemu-kvm.x86_64 2:0.12.1.2-2.355.0.1.el6.centos.2
  virt-manager.x86_64 0:0.9.0-18.el6               virt-top.x86_64 0:1.0.4-3.15.el6      
        virt-viewer.x86_64 0:0.5.2-18.el6_4.2
  
Modify libvirt configuration:
	Edit /etc/libvirt/libvirtd.conf
		listen_tls = 0
		listen_tcp = 1
		tcp_port = 16059
		auth_tcp = "none"
		mdns_adv = 0

	Edit /etc/sysconfig/libvirtd
		LIBVIRTD_ARGS="--listen"
  
Start libvirtd
   # /etc/init.d/libvirtd start
   # /etc/init.d/libvirtd status

Verify installation:
  # virsh capabilities 
  - Should list two guest tags with os_type hvm
  
Build and install openvswitch:
  Install build requirements:
     # yum install rpmdevtools openssl-devel kernel-devel gcc redhat-rpm-config
	 
  Build packages:
     # mkdir -p ~/rpmbuild/SOURCES
     # curl -O http://openvswitch.org/releases/openvswitch-1.9.0.tar.gz
	 # cp openvswitch-1.9.0.tar.gz ~/rpmbuild/SOURCES
	 # cp centos64-openvswitch.patch ~/rpmbuild/SOURCES
	 # tar -xzf openvswitch-1.9.0.tar.gz
	 # cd openvswitch-1.9.0
	 # patch -p1 < ~/rpmbuild/SOURCES/centos64-openvswitch.patch
	 # rpmbuild -bb rhel/openvswitch.spec
	 # rpmbuild -bb -D "kversion `uname -r`" rhel/openvswitch-kmod-rhel6.spec
	 
  Install openvswitch:
    # yum install ~/rpmbuild/RPMS/x86_64/kmod-openvswitch-1.9.0-1.el6.x86_64.rpm ~/rpmbuild/RPMS/x86_64/openvswitch-1.9.0-1.x86_64.rpm
	# echo 'blacklist bridge' >> /etc/modprobe.d/blacklist.conf
	# reboot
	
  Verify installation:
    # lsmod |grep openvswitch
	# ovs-vsctl -V
	
Network design:
  cloudbr0 (Management, Storage)
    ip: 172.16.10.10/24
	gateway: 172.16.10.1
	eth0 (physical port, no vlans)
  cloudbr1 (Guest, Public)
    eth1 (physical port, vlan trunk)
	ip: none
	
Configure network interfaces:
   /etc/sysconfig/network-scripts/ifcfg-eth0
	DEVICE=eth0
	BOOTPROTO=none
	IPV6INIT=no
	NM_CONTROLLED=no
	ONBOOT=yes
	TYPE=OVSPort
	DEVICETYPE=ovs
	OVS_BRIDGE=cloudbr0

   /etc/sysconfig/network-scripts/ifcfg-eth1
	DEVICE=eth1
	BOOTPROTO=none
	IPV6INIT=no
	NM_CONTROLLED=no
	ONBOOT=yes
	TYPE=OVSPort
	DEVICETYPE=ovs
	OVS_BRIDGE=cloudbr1

   /etc/sysconfig/network-scripts/ifcfg-cloudbr0
	DEVICE=cloudbr0
	ONBOOT=yes
	DEVICETYPE=ovs
	TYPE=OVSBridge
	BOOTPROTO=static
	IPADDR=172.16.10.10
	GATEWAY=172.16.10.1
	NETMASK=255.255.255.0
	HOTPLUG=no

   /etc/sysconfig/network-scripts/ifcfg-cloudbr1
	DEVICE=cloudbr1
	ONBOOT=yes
	DEVICETYPE=ovs
	TYPE=OVSBridge
	BOOTPROTO=none
	HOTPLUG=no

   /etc/sysconfig/network
    NETWORKING=yes
	HOSTNAME=testkvm1
	GATEWAY=172.10.10.1
	
Install cloudstack-agent:
	# yum install cloudstack-agent
	
Edit /etc/cloudstack/agent/agent.properties
    network.bridge.type=openvswitch
    libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver
	
Now add the host to cloudstack.
	

  
================  centos64-openvswitch.patch =====================  
diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h   2013-02-26
21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h 2013-04-25 10:45:09.942027933
+0200
@@ -4,16 +4,4 @@
 #include <linux/version.h>
 #include_next <linux/etherdevice.h>

-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,36)
-static inline void eth_hw_addr_random(struct net_device *dev)
-{
-       random_ether_addr(dev->dev_addr);
-}
-#elif LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0)
-static inline void eth_hw_addr_random(struct net_device *dev)
-{
-       dev_hw_addr_random(dev, dev->dev_addr);
-}
-#endif
-
 #endif
diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h       2013-02-26
21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h     2013-04-25 10:44:33.270023182
+0200
@@ -55,38 +55,4 @@
 #define VLAN_TAG_PRESENT       VLAN_CFI_MASK
 #endif

-#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0)
-static inline void vlan_set_encap_proto(struct sk_buff *skb, struct vlan_hdr *vhdr)
-{
-       __be16 proto;
-       unsigned char *rawp;
-
-       /*
-        * Was a VLAN packet, grab the encapsulated protocol, which the layer
-        * three protocols care about.
-        */
-
-       proto = vhdr->h_vlan_encapsulated_proto;
-       if (ntohs(proto) >= 1536) {
-               skb->protocol = proto;
-               return;
-       }
-
-       rawp = skb->data;
-       if (*(unsigned short *) rawp == 0xFFFF)
-               /*
-                * This is a magic hack to spot IPX packets. Older Novell
-                * breaks the protocol design and runs IPX over 802.3 without
-                * an 802.2 LLC layer. We look for FFFF which isn't a used
-                * 802.2 SSAP/DSAP. This won't work for fault tolerant netware
-                * but does for the rest.
-                */
-               skb->protocol = htons(ETH_P_802_3);
-       else
-               /*
-                * Real 802.2 LLC
-                */
-               skb->protocol = htons(ETH_P_802_2);
-}
-#endif
 #endif /* linux/if_vlan.h wrapper */
diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h        2013-02-26
21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h      2013-04-25 10:43:21.167021697
+0200
@@ -245,10 +245,4 @@
 }
 #endif

-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,40)
-static inline void skb_reset_mac_len(struct sk_buff *skb)
-{
-       skb->mac_len = skb->network_header - skb->mac_header;
-}
-#endif
 #endif
diff -ru openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec
--- openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec    2013-02-26 21:25:52.000000000
+0100
+++ openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec  2013-04-25 10:40:41.039023915 +0200
@@ -22,6 +22,7 @@
 Source0:        %{oname}-%{version}.tar.gz
 BuildRoot:      %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
 BuildRequires:  %kernel_module_package_buildreqs
+Patch0:                centos64-openvswitch.patch

 # Without this we get an empty openvswitch-debuginfo package (whose name
 # conflicts with the openvswitch-debuginfo package for OVS userspace).
@@ -43,6 +44,7 @@
 %prep

 %setup -n %{oname}-%{version}
+%patch0 -p1

 %build
 for flavor in %flavors_to_build; do
================  centos64-openvswitch.patch =====================


> -----Original Message-----
> From: Angeline Shen [mailto:Angeline.Shen@citrix.com]
> Sent: Thursday, April 25, 2013 5:31 AM
> To: '<dev@cloudstack.apache.org>'; Hugo Trippaers; Sheng Yang; Edison Su
> Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
> 
> Hugo:
> 
> For your configuration example  :
> 
> 
> # The physical interface: eth1
> DEVICE=eth1
> BOOTPROTO=none
> HWADDR=BC:30:5B:D4:16:3C
> IPV6INIT=no
> #MTU=1500
> NM_CONTROLLED=no
> ONBOOT=yes
> TYPE=OVSPort
> DEVICETYPE=ovs
> OVS_BRIDGE=br0
> 
> 
> # The bridge interface: br0
> DEVICE=br0
> ONBOOT=yes
> DEVICETYPE=ovs
> TYPE=OVSBridge
> BOOTPROTO=static
> IPADDR= 10.223.58.195
> GATEWAY = 10.223.58.193
> NETMASK=255.255.255.192
> HOTPLUG=no
> 
> QUESTION: will commands to configure openvswitch network interfaces be
> as follows:
> 
> ovs-vsctl add-br br0
> ovs-vsctl add-port br0 eth1
>

The redhat network configuration should take care of creating all the bridges and linking
the ports. But yes the configuration above should have the same effect as these two commands.


> 
> Thanks
> 
> -----Original Message-----
> From: Angeline Shen
> Sent: Wednesday, April 24, 2013 1:13 PM
> To: <dev@cloudstack.apache.org>; Hugo Trippaers; Sheng Yang; Edison Su
> Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
> 
> Hugo:
> 
> Please review my updates to
> https://issues.apache.org/jira/browse/CLOUDSTACK-101 .
> 
> The latest experiment I conducted per your instructions  is as follows:
> 
>  ovs  Host    10.223.58.195          gateway     10.223.58.193          other host on
> same gateway but not in ovs setup   10.223.58.194
> 
> On ovs host 10.223.58.195 , After I changed /etc/sysconfig/network-
> scripts/ifcfg-eth1 and  /etc/sysconfig/network-scripts/ifcfg-cloudbr to your
> instructions, service network restart ,
> 
> the ovs host can now ONLY ping its own gateway 10.223.58.193 and any host
> in same subnet, but CANNOT reach anything beyond the gateway
> 10.223.58.193.
> 
> 1. /etc/sysconfig/network-scripts/ifcfg-eth1:
> 
> DEVICE=eth1
> BOOTPROTO=none
> HWADDR=BC:30:5B:D4:16:3C
> IPV6INIT=no
> #MTU=1500
> NM_CONTROLLED=no
> ONBOOT=yes
> #TYPE=Ethernet
> TYPE=OVSPort
> DEVICETYPE=ovs
> OVS_BRIDGE=cloudbr
> #UUID="0142ba15-de11-499c-a1ba-82837b30f890"
> #IPADDR=10.223.58.195
> #NETMASK=255.255.255.192
> #GATEWAY=10.223.58.193
> #DNS1=10.223.110.254
> #USERCTL=yes
> #IPV4_FAILURE_FATAL=yes
> #DEFROUTE=yes
> #NAME="System eth1"
> 
> 2. /etc/sysconfig/network-scripts/ifcfg-cloudbr:
> 
> DEVICE=cloudbr
> ONBOOT=yes
> DEVICETYPE=ovs
> TYPE=OVSBridge
> BOOTPROTO=static
> IPADDR=10.223.58.195
> GATEWAY=10.23.58.193
> NETMASK=255.255.255.0
> HOTPLUG=no
> 
> 3. On host 10.223.58.195, service network restart
> 
> 
> 4. on host 10.223.58.195:
> 
> [root@Rack3Host16 ~]# ip route
> 10.223.58.0/24 dev cloudbr proto kernel scope link src 10.223.58.195
> 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
> 169.254.0.0/16 dev eth1 scope link metric 1002
> 169.254.0.0/16 dev cloudbr scope link metric 1015
> 
> 5. It can reach its gateway & other hosts on same gateway:
> 
> [root@Rack3Host16 ~]# ping 10.223.58.193 PING 10.223.58.193
> (10.223.58.193) 56(84) bytes of data.
> 64 bytes from 10.223.58.193: icmp_seq=1 ttl=64 time=7.43 ms
> 64 bytes from 10.223.58.193: icmp_seq=2 ttl=64 time=2.82 ms
> 64 bytes from 10.223.58.193: icmp_seq=3 ttl=64 time=1.06 ms ^C
> --- 10.223.58.193 ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2364ms rtt
> min/avg/max/mdev = 1.061/3.771/7.430/2.685 ms
> [root@Rack3Host16 ~]# ping 10.223.58.194 PING 10.223.58.194
> (10.223.58.194) 56(84) bytes of data.
> 64 bytes from 10.223.58.194: icmp_seq=1 ttl=64 time=0.399 ms
> 64 bytes from 10.223.58.194: icmp_seq=2 ttl=64 time=0.181 ms ^C
> --- 10.223.58.194 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1446ms rtt
> min/avg/max/mdev = 0.181/0.290/0.399/0.109 ms
> 
> [root@Rack3Host16 ~]# traceroute 10.223.58.194 traceroute to 10.223.58.194
> (10.223.58.194), 30 hops max, 60 byte packets
>  1 10.223.58.194 (10.223.58.194) 0.922 ms 0.851 ms 0.830 ms
> 
> [root@Rack3Host16 ~]# traceroute 10.223.58.193 traceroute to 10.223.58.193
> (10.223.58.193), 30 hops max, 60 byte packets
>  1 10.223.58.193 (10.223.58.193) 7.017 ms 12.506 ms 7.675 ms
> 
> 6. host CANNOT reach anything outside of its subnet:
> 
> [root@Rack3Host16 ~]# ping 10.223.195.114
> connect: Network is unreachable
> 
> [root@Rack3Host16 ~]# ping www.google.com
> ping: unknown host www.google.com
> 
> [root@Rack3Host16 ~]# ping 8.8.8.8
> connect: Network is unreachable
> 
> [root@Rack3Host16 ~]# traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 30
> hops max, 60 byte packets
> connect: Network is unreachable
> 
> 7. [root@Rack3Host16 ~]# ifconfig
> cloudbr Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C
>           inet addr:10.223.58.195 Bcast:10.223.58.255 Mask:255.255.255.0
>           inet6 addr: fc00:2::be30:5bff:fed4:163c/64 Scope:Global
>           inet6 addr: fe80::68f7:52ff:fe03:a744/64 Scope:Link
>           UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
>           RX packets:46945 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:549 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:39235509 (37.4 MiB) TX bytes:102773 (100.3 KiB)
> 
> cloud0 Link encap:Ethernet HWaddr AA:81:04:5F:5E:4A
>           inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
>           inet6 addr: fe80::a881:4ff:fe5f:5e4a/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 b) TX bytes:4201 (4.1 KiB)
> 
> eth1 Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C
>           inet6 addr: fe80::be30:5bff:fed4:163c/64 Scope:Link
>           UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
>           RX packets:48123 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:532 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:39518352 (37.6 MiB) TX bytes:101960 (99.5 KiB)
>           Interrupt:16 Memory:da000000-da012800
> 
> lo Link encap:Local Loopback
>           inet addr:127.0.0.1 Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING MTU:16436 Metric:1
>           RX packets:340 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:340 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:46264 (45.1 KiB) TX bytes:46264 (45.1 KiB)
> 
> 8. [root@Rack3Host16 ~]# brctl show
> bridge name bridge id STP enabled interfaces
> cloud0 /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory 0000.aa81045f5e4a no
> cloudbr /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory 0000.bc305bd4163c
> no
> 
> 
> -----Original Message-----
> From: Sheng Yang [mailto:sheng@yasker.org]
> Sent: Wednesday, April 24, 2013 12:59 PM
> To: Hugo Trippaers
> Cc: <dev@cloudstack.apache.org>
> Subject: Re: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
> 
> Thank you Hugo!
> 
> --Sheng
> 
> 
> On Wed, Apr 24, 2013 at 12:16 PM, Hugo Trippaers <
> HTrippaers@schubergphilis.com> wrote:
> 
> >  Hey Sheng,
> >
> >  Sure thing, I'm trying to help Angeline already, but it's quite a
> > complex setup.
> >
> >  I'll try to write down some detailed instructions.
> >
> >  Cheers,
> >
> >  Hugo
> >
> > Sent from my iPhone
> >
> > On 24 apr. 2013, at 19:56, "Sheng Yang" <sheng@yasker.org> wrote:
> >
> >   Hi Hugo,
> >
> >  Could you help with KVM OVS setup? I cannot find much information on
> > the our wiki about OVS on KVM.
> >
> >  --Sheng
> >
> >  On Fri, Apr 19, 2013 at 8:14 PM, Angeline Shen
> <Angeline.Shen@citrix.com>wrote:
> >
> >> ASF 4.1  cloudstack  with openvswitch  support  for   RHEL 6.3   KVM test
> >>  configuration setup:
> >>
> >> After completing
> >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on-
> cent
> >> os-6-3-minimal/  to setup RHEL 6.3 openvswitch, encountered following
> >> problems:
> >>
> >> 1. In this scenario, after service openvswitch start, host is still
> >> up running but completely lost connection to network.
> >>
> >> [root@Rack3Host16 /]# service openvswitch start Inserting brcompat
> >> module [ OK ] Starting ovsdb-server [ OK ] Configuring Open vSwitch
> >> system IDs [ OK ] Starting ovs-vswitchd [ OK ] Starting ovs-brcompatd
> >> [ OK ] iptables already has a rule for gre, not explicitly enabling.
> >> [root@Rack3Host16 /]#
> >>
> >>
> >> 2. content of various files:
> >>
> >> /etc/sysconfig/network-scripts/ifcfg-ovsbr1 :
> >> DEVICE=ovsbr1
> >> ONBOOT=yes
> >> DEVICETYPE=ovs
> >> TYPE=OVSBridge
> >> BOOTPROTO=static
> >> IPADDR=10.223.58.195
> >> GATEWAY=10.223.58.193
> >> NETMASK=255.255.255.0
> >> HOTPLUG=no
> >>
> >>
> >> QUESTION:    Is
> >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on-
> centos-6-3-minimal/
> >>         Valid configuration instructions for   openvswitch   RHEL 6.3
> >>  or are there other updated  correct configuration steps?
> >>
> >> Thanks
> >>
> >>
> >

Mime
View raw message