Return-Path: X-Original-To: apmail-incubator-cloudstack-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-cloudstack-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B28219C8F for ; Mon, 18 Jun 2012 10:21:51 +0000 (UTC) Received: (qmail 37433 invoked by uid 500); 18 Jun 2012 10:21:51 -0000 Delivered-To: apmail-incubator-cloudstack-dev-archive@incubator.apache.org Received: (qmail 37107 invoked by uid 500); 18 Jun 2012 10:21:47 -0000 Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cloudstack-dev@incubator.apache.org Delivered-To: mailing list cloudstack-dev@incubator.apache.org Received: (qmail 37078 invoked by uid 99); 18 Jun 2012 10:21:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jun 2012 10:21:46 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Salvatore.Orlando@eu.citrix.com designates 62.200.22.115 as permitted sender) Received: from [62.200.22.115] (HELO SMTP.EU.CITRIX.COM) (62.200.22.115) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jun 2012 10:21:39 +0000 X-IronPort-AV: E=Sophos;i="4.75,791,1330905600"; d="scan'208";a="13073798" Received: from lonpmailmx02.citrite.net ([10.30.203.163]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 18 Jun 2012 10:21:18 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX02.citrite.net ([10.30.203.163]) with mapi; Mon, 18 Jun 2012 11:21:18 +0100 From: Salvatore Orlando To: "cloudstack-dev@incubator.apache.org" Date: Mon, 18 Jun 2012 11:21:16 +0100 Subject: RE: Advice on SSVM network interfaces Thread-Topic: Advice on SSVM network interfaces Thread-Index: Ac1LIKa86F54uc+BSTaoWaVfJzPCsQAB0ygQAAG8oaAAAID+IACCC29A Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hi,=20 I've created an entry in Jira for this issue: http://bugs.cloudstack.org/br= owse/CS-15308 dumps for all the VIFs of the SSVM are attached to the Jira entry itself. An Extract from the traces, obtained while trying to ping the address of se= condary storage NFS share, is reported below. Please note that the secondary storage NFS share is on a distinct network, = and the gateway configured for the pod (192.168.0.1) NATs to this network.= =20 eth1 11:49:29.082189 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.103 tell 192.168.0.1, length 46 11:49:29.082202 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.3.103 is= -at 06:c1:7c:00:00:04, length 28 11:49:29.226796 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:30.227001 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 eth2 11:49:25.218750 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 1280, le= ngth 64 11:49:25.220138 IP (tos 0x0, ttl 63, id 3370, offset 0, flags [none], proto= ICMP (1), length 84) 10.70.177.15 > 192.168.3.110: ICMP echo reply, id 33826, seq 1280, leng= th 64 11:49:25.225618 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:26.220070 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 1536, le= ngth 64 11:49:26.221735 IP (tos 0x0, ttl 63, id 3371, offset 0, flags [none], proto= ICMP (1), length 84) 10.70.177.15 > 192.168.3.110: ICMP echo reply, id 33826, seq 1536, leng= th 64 11:49:26.225690 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:27.221391 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 1792, le= ngth 64 11:49:27.222894 IP (tos 0x0, ttl 63, id 3372, offset 0, flags [none], proto= ICMP (1), length 84) 10.70.177.15 > 192.168.3.110: ICMP echo reply, id 33826, seq 1792, leng= th 64 11:49:27.225831 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:28.222711 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 2048, le= ngth 64 11:49:28.225105 IP (tos 0x0, ttl 63, id 3373, offset 0, flags [none], proto= ICMP (1), length 84) 10.70.177.15 > 192.168.3.110: ICMP echo reply, id 33826, seq 2048, leng= th 64 11:49:29.224031 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 2304, le= ngth 64 11:49:29.226845 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:30.225353 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP= (1), length 84) 192.168.3.110 > 10.70.177.15: ICMP echo request, id 33826, seq 2560, le= ngth 64 11:49:30.227048 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 eth3 11:49:29.226817 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 11:49:29.687318 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.9.10 tell 192.168.9.12, length 46 11:49:30.227018 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.16= 8.3.110 tell 192.168.0.1, length 46 > -----Original Message----- > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com] > Sent: 15 June 2012 20:58 > To: cloudstack-dev@incubator.apache.org > Subject: RE: Advice on SSVM network interfaces >=20 > Salvatore, >=20 > I guess the attachment is discarded by mail server, could you please post= the > content in mail? >=20 > Please file a bug for this. >=20 >=20 > Regards, > Anthony >=20 > > -----Original Message----- > > From: Salvatore Orlando [mailto:Salvatore.Orlando@eu.citrix.com] > > Sent: Friday, June 15, 2012 12:49 PM > > To: cloudstack-dev@incubator.apache.org > > Subject: RE: Advice on SSVM network interfaces > > > > Anthony, > > > > At first I too was suspecting the ARP reply was sent on eth1 and then > > discarded by the gateway because the SRC MAC address of the frame did > > not match the MAC in the ARP payload. > > However, I've not seen any ARP reply on eth1. I'm attaching the dumps > > so you can have a look at them (I don't know whether they'll be > > mangled or not, though). > > > > I agree this might be a bug. My perspective is that there's no reason > > for having several network interfaces unless distinct labels are being > > used for management/public/storage network. > > Is it ok for you if I report an issue on bugs.cloudstack.org and > > assign it to you? > > > > Regards, > > Salvatore > > > > > -----Original Message----- > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com] > > > Sent: 15 June 2012 19:59 > > > To: cloudstack-dev@incubator.apache.org > > > Subject: RE: Advice on SSVM network interfaces > > > > > > Hi Salvatore, > > > > > > From your description, the ARP response is sent out through eth1, > > some > > > switches may drop this kind of package, it expects to receive ARP > > response > > > from the same port ARP request sent out. > > > > > > I think it is a bug, in this case, CloudStack should not configure > > eth2 and eth3 > > > for SSVM, if SSVM does need several IPs, all IPs should be > > > configured > > on eth1 > > > if they are in the same subnet. > > > > > > > > > > > > Regards, > > > Anthony > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Salvatore Orlando [mailto:Salvatore.Orlando@eu.citrix.com] > > > > Sent: Friday, June 15, 2012 11:00 AM > > > > To: cloudstack-dev@incubator.apache.org > > > > Subject: Advice on SSVM network interfaces > > > > > > > > Hi, > > > > > > > > We have a test environment where a basic zone is deployed. > > > > Both system VMs and guest addresses are in the 192.168.0.0/16 > > subnet, > > > > even if with distinct IP ranges. > > > > > > > > We noticed that the SSVM is unable to download templates, as the > > > > connection over the public interface (eth2) is suddenly dropped > > (see > > > > attached dump). > > > > As it can be seen from the dump the connection drops because the > > SSVM > > > > fails to answer to ARP requests from the gateway on eth2. > > > > ARP requests sent to eth2's address fail also from other machines > > in > > > > the same network. > > > > > > > > Here are the relevant configuration info from the SSVM: > > > > > > > > root@s-3-VM:~# ip addr show > > > > 1: lo: mtu 16436 qdisc noqueue state > > > UNKNOWN > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > inet 127.0.0.1/8 scope host lo > > > > inet6 ::1/128 scope host > > > > valid_lft forever preferred_lft forever > > > > 2: eth0: mtu 1500 qdisc > > > pfifo_fast > > > > state UP qlen 1000 > > > > link/ether 0e:00:a9:fe:02:5c brd ff:ff:ff:ff:ff:ff > > > > inet 169.254.2.92/16 brd 169.254.255.255 scope global eth0 > > > > inet6 fe80::c00:a9ff:fefe:25c/64 scope link > > > > valid_lft forever preferred_lft forever > > > > 3: eth1: mtu 1500 qdisc > > > pfifo_fast > > > > state UP qlen 1000 > > > > link/ether 06:de:1e:00:00:03 brd ff:ff:ff:ff:ff:ff > > > > inet 192.168.3.102/16 brd 192.168.255.255 scope global eth1 > > > > inet6 fe80::4de:1eff:fe00:3/64 scope link > > > > valid_lft forever preferred_lft forever > > > > 4: eth2: mtu 1500 qdisc > > > pfifo_fast > > > > state UP qlen 1000 > > > > link/ether 06:5d:f6:00:00:0b brd ff:ff:ff:ff:ff:ff > > > > inet 192.168.3.110/16 brd 192.168.255.255 scope global eth2 > > > > inet6 fe80::45d:f6ff:fe00:b/64 scope link > > > > valid_lft forever preferred_lft forever > > > > 5: eth3: mtu 1500 qdisc > > > pfifo_fast > > > > state UP qlen 1000 > > > > link/ether 06:93:c6:00:00:04 brd ff:ff:ff:ff:ff:ff > > > > inet 192.168.3.103/16 brd 192.168.255.255 scope global eth3 > > > > inet6 fe80::493:c6ff:fe00:4/64 scope link > > > > valid_lft forever preferred_lft forever > > > > > > > > > > > > root@s-3-VM:~# ip route > > > > 169.254.0.0/16 dev eth0 proto kernel scope link src > > > > 169.254.2.92 > > > > 192.168.0.0/16 dev eth1 proto kernel scope link src > > 192.168.3.102 > > > > 192.168.0.0/16 dev eth2 proto kernel scope link src > > 192.168.3.110 > > > > 192.168.0.0/16 dev eth3 proto kernel scope link src > > 192.168.3.103 > > > > default via 192.168.0.1 dev eth2 > > > > > > > > > > > > root@s-3-VM:~# sysctl -a | grep ipv4.conf.*.arp > > > > error: permission denied on key 'net.ipv4.route.flush' > > > > net.ipv4.conf.all.proxy_arp =3D 0 > > > > net.ipv4.conf.all.arp_filter =3D 0 > > > > net.ipv4.conf.all.arp_announce =3D 2 net.ipv4.conf.all.arp_ignore = =3D > > > > 2 net.ipv4.conf.all.arp_accept =3D 0 net.ipv4.conf.all.arp_notify = =3D > > > > 0 net.ipv4.conf.default.proxy_arp =3D 0 > > > > net.ipv4.conf.default.arp_filter =3D 0 > > > > net.ipv4.conf.default.arp_announce =3D 2 > > > > net.ipv4.conf.default.arp_ignore =3D 2 > > net.ipv4.conf.default.arp_accept > > > > =3D 0 net.ipv4.conf.default.arp_notify =3D 0 > > > > net.ipv4.conf.lo.proxy_arp > > =3D > > > > 0 net.ipv4.conf.lo.arp_filter =3D 0 net.ipv4.conf.lo.arp_announce = =3D > > > > 2 net.ipv4.conf.lo.arp_ignore =3D 2 net.ipv4.conf.lo.arp_accept =3D= 0 > > > > net.ipv4.conf.lo.arp_notify =3D 0 net.ipv4.conf.eth0.proxy_arp =3D = 0 > > > > net.ipv4.conf.eth0.arp_filter =3D 0 net.ipv4.conf.eth0.arp_announce > > > > =3D > > 2 > > > > net.ipv4.conf.eth0.arp_ignore =3D 2 net.ipv4.conf.eth0.arp_accept = =3D > > > > 0 net.ipv4.conf.eth0.arp_notify =3D 0 net.ipv4.conf.eth1.proxy_arp = =3D > > > > 0 net.ipv4.conf.eth1.arp_filter =3D 0 > > > > net.ipv4.conf.eth1.arp_announce =3D > > 2 > > > > net.ipv4.conf.eth1.arp_ignore =3D 2 net.ipv4.conf.eth1.arp_accept = =3D > > > > 0 net.ipv4.conf.eth1.arp_notify =3D 0 net.ipv4.conf.eth2.proxy_arp = =3D > > > > 0 net.ipv4.conf.eth2.arp_filter =3D 0 > > > > net.ipv4.conf.eth2.arp_announce =3D > > 2 > > > > net.ipv4.conf.eth2.arp_ignore =3D 2 net.ipv4.conf.eth2.arp_accept = =3D > > > > 0 net.ipv4.conf.eth2.arp_notify =3D 0 net.ipv4.conf.eth3.proxy_arp = =3D > > > > 0 net.ipv4.conf.eth3.arp_filter =3D 0 > > > > net.ipv4.conf.eth3.arp_announce =3D > > 2 > > > > net.ipv4.conf.eth3.arp_ignore =3D 2 net.ipv4.conf.eth3.arp_accept = =3D > > > > 0 net.ipv4.conf.eth3.arp_notify =3D 0 > > > > > > > > The behaviour actually is exactly the same one would expect if > > > > arp_filter is enabled on the interfaces, but the flag is clearly > > set > > > > to 0. Also setting arp_ignore to 0 does not cause the expected arp > > > > flux problem, as replies are sent only from the first virtual > > > > interface (eth1). In a way, it looks like as there are policies > > > > enforced through arptables, but it seems the module is not loaded, > > nor > > > > the userspace utility is available on the SSVM. > > > > > > > > Of course, changing the order in the route table as follows, ie > > > > putting > > > > eth2 before eth1 for 192.168.0.0/16, solves the issue. > > > > > > > > 169.254.0.0/16 dev eth0 proto kernel scope link src > > > > 169.254.2.92 > > > > 192.168.0.0/16 dev eth2 proto kernel scope link src > > 192.168.3.110 > > > > 192.168.0.0/16 dev eth1 proto kernel scope link src > > 192.168.3.102 > > > > 192.168.0.0/16 dev eth3 proto kernel scope link src > > 192.168.3.103 > > > > default via 192.168.0.1 dev eth2 > > > > > > > > Quite interestingly, after this change ARP requests to eth2 are > > > > honoured by the SSVM even after it is rebooted, and even if the > > > > relevant ARP cache entry in the gateway is removed. Of course, > > > > this > > is > > > > not the case when the SSVM is destroyed, as the new SSVM will have > > a > > > > different MAC address for every interface. > > > > > > > > It is also interesting noting that in another setup, where we > > > > configured an advanced zone, this problem does not occur. Even the > > > > SSVM is deployed in an adv zone, the network configuration of the > > SSVM > > > > is very similar. > > > > > > > > It would be great if you can provide some advice for debugging > > > > this issue, or share similar experiences. > > > > > > > > Regards, > > > > Salvatore