incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Xu <Xuefei...@citrix.com>
Subject RE: Advice on SSVM network interfaces
Date Fri, 15 Jun 2012 19:58:27 GMT
Salvatore,

I guess the attachment is discarded by mail server, could you please post the content in mail?

Please file a bug for this.


Regards,
Anthony

> -----Original Message-----
> From: Salvatore Orlando [mailto:Salvatore.Orlando@eu.citrix.com]
> Sent: Friday, June 15, 2012 12:49 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Advice on SSVM network interfaces
> 
> Anthony,
> 
> At first I too was suspecting the ARP reply was sent on eth1 and then
> discarded by the gateway because the SRC MAC address of the frame did
> not match the MAC in the ARP payload.
> However, I've not seen any ARP reply on eth1. I'm attaching the dumps
> so you can have a look at them (I don't know whether they'll be mangled
> or not, though).
> 
> I agree this might be a bug. My perspective is that there's no reason
> for having several network interfaces unless distinct labels are being
> used for management/public/storage network.
> Is it ok for you if I report an issue on bugs.cloudstack.org and assign
> it to you?
> 
> Regards,
> Salvatore
> 
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: 15 June 2012 19:59
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Advice on SSVM network interfaces
> >
> > Hi Salvatore,
> >
> > From your description, the ARP response is sent out through eth1,
> some
> > switches may drop this kind of package, it expects to receive ARP
> response
> > from the same port ARP request sent out.
> >
> > I think it is a bug, in this case, CloudStack should not configure
> eth2 and eth3
> > for SSVM, if SSVM does need several IPs, all IPs should be configured
> on eth1
> > if they are in the same subnet.
> >
> >
> >
> > Regards,
> > Anthony
> >
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Salvatore Orlando [mailto:Salvatore.Orlando@eu.citrix.com]
> > > Sent: Friday, June 15, 2012 11:00 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Advice on SSVM network interfaces
> > >
> > > Hi,
> > >
> > > We have a test environment where a basic zone is deployed.
> > > Both system VMs and guest addresses are in the 192.168.0.0/16
> subnet,
> > > even if with distinct IP ranges.
> > >
> > > We noticed that the SSVM is unable to download templates, as the
> > > connection over the public interface (eth2) is suddenly dropped
> (see
> > > attached dump).
> > > As it can be seen from the dump the connection drops because the
> SSVM
> > > fails to answer to ARP requests from the gateway on eth2.
> > > ARP requests sent to eth2's address fail also from other machines
> in
> > > the same network.
> > >
> > > Here are the relevant configuration info from the SSVM:
> > >
> > > root@s-3-VM:~# ip addr show
> > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> > UNKNOWN
> > >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > >     inet 127.0.0.1/8 scope host lo
> > >     inet6 ::1/128 scope host
> > >        valid_lft forever preferred_lft forever
> > > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > pfifo_fast
> > > state UP qlen 1000
> > >     link/ether 0e:00:a9:fe:02:5c brd ff:ff:ff:ff:ff:ff
> > >     inet 169.254.2.92/16 brd 169.254.255.255 scope global eth0
> > >     inet6 fe80::c00:a9ff:fefe:25c/64 scope link
> > >        valid_lft forever preferred_lft forever
> > > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > pfifo_fast
> > > state UP qlen 1000
> > >     link/ether 06:de:1e:00:00:03 brd ff:ff:ff:ff:ff:ff
> > >     inet 192.168.3.102/16 brd 192.168.255.255 scope global eth1
> > >     inet6 fe80::4de:1eff:fe00:3/64 scope link
> > >        valid_lft forever preferred_lft forever
> > > 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > pfifo_fast
> > > state UP qlen 1000
> > >     link/ether 06:5d:f6:00:00:0b brd ff:ff:ff:ff:ff:ff
> > >     inet 192.168.3.110/16 brd 192.168.255.255 scope global eth2
> > >     inet6 fe80::45d:f6ff:fe00:b/64 scope link
> > >        valid_lft forever preferred_lft forever
> > > 5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > pfifo_fast
> > > state UP qlen 1000
> > >     link/ether 06:93:c6:00:00:04 brd ff:ff:ff:ff:ff:ff
> > >     inet 192.168.3.103/16 brd 192.168.255.255 scope global eth3
> > >     inet6 fe80::493:c6ff:fe00:4/64 scope link
> > >        valid_lft forever preferred_lft forever
> > >
> > >
> > > root@s-3-VM:~# ip route
> > > 169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.2.92
> > > 192.168.0.0/16 dev eth1  proto kernel  scope link  src
> 192.168.3.102
> > > 192.168.0.0/16 dev eth2  proto kernel  scope link  src
> 192.168.3.110
> > > 192.168.0.0/16 dev eth3  proto kernel  scope link  src
> 192.168.3.103
> > > default via 192.168.0.1 dev eth2
> > >
> > >
> > > root@s-3-VM:~# sysctl -a | grep ipv4.conf.*.arp
> > > error: permission denied on key 'net.ipv4.route.flush'
> > > net.ipv4.conf.all.proxy_arp = 0
> > > net.ipv4.conf.all.arp_filter = 0
> > > net.ipv4.conf.all.arp_announce = 2
> > > net.ipv4.conf.all.arp_ignore = 2
> > > net.ipv4.conf.all.arp_accept = 0
> > > net.ipv4.conf.all.arp_notify = 0
> > > net.ipv4.conf.default.proxy_arp = 0
> > > net.ipv4.conf.default.arp_filter = 0
> > > net.ipv4.conf.default.arp_announce = 2
> > > net.ipv4.conf.default.arp_ignore = 2
> net.ipv4.conf.default.arp_accept
> > > = 0 net.ipv4.conf.default.arp_notify = 0 net.ipv4.conf.lo.proxy_arp
> =
> > > 0 net.ipv4.conf.lo.arp_filter = 0 net.ipv4.conf.lo.arp_announce = 2
> > > net.ipv4.conf.lo.arp_ignore = 2 net.ipv4.conf.lo.arp_accept = 0
> > > net.ipv4.conf.lo.arp_notify = 0 net.ipv4.conf.eth0.proxy_arp = 0
> > > net.ipv4.conf.eth0.arp_filter = 0 net.ipv4.conf.eth0.arp_announce =
> 2
> > > net.ipv4.conf.eth0.arp_ignore = 2 net.ipv4.conf.eth0.arp_accept = 0
> > > net.ipv4.conf.eth0.arp_notify = 0 net.ipv4.conf.eth1.proxy_arp = 0
> > > net.ipv4.conf.eth1.arp_filter = 0 net.ipv4.conf.eth1.arp_announce =
> 2
> > > net.ipv4.conf.eth1.arp_ignore = 2 net.ipv4.conf.eth1.arp_accept = 0
> > > net.ipv4.conf.eth1.arp_notify = 0 net.ipv4.conf.eth2.proxy_arp = 0
> > > net.ipv4.conf.eth2.arp_filter = 0 net.ipv4.conf.eth2.arp_announce =
> 2
> > > net.ipv4.conf.eth2.arp_ignore = 2 net.ipv4.conf.eth2.arp_accept = 0
> > > net.ipv4.conf.eth2.arp_notify = 0 net.ipv4.conf.eth3.proxy_arp = 0
> > > net.ipv4.conf.eth3.arp_filter = 0 net.ipv4.conf.eth3.arp_announce =
> 2
> > > net.ipv4.conf.eth3.arp_ignore = 2 net.ipv4.conf.eth3.arp_accept = 0
> > > net.ipv4.conf.eth3.arp_notify = 0
> > >
> > > The behaviour actually is exactly the same one would expect if
> > > arp_filter is enabled on the interfaces, but the flag is clearly
> set
> > > to 0. Also setting arp_ignore to 0 does not cause the expected arp
> > > flux problem, as replies are sent only from the first virtual
> > > interface (eth1). In a way, it looks like as there are policies
> > > enforced through arptables, but it seems the module is not loaded,
> nor
> > > the userspace utility is available on the SSVM.
> > >
> > > Of course, changing the order in the route table as follows, ie
> > > putting
> > > eth2 before eth1 for 192.168.0.0/16, solves the issue.
> > >
> > > 169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.2.92
> > > 192.168.0.0/16 dev eth2  proto kernel  scope link  src
> 192.168.3.110
> > > 192.168.0.0/16 dev eth1  proto kernel  scope link  src
> 192.168.3.102
> > > 192.168.0.0/16 dev eth3  proto kernel  scope link  src
> 192.168.3.103
> > > default via 192.168.0.1 dev eth2
> > >
> > > Quite interestingly, after this change ARP requests to eth2 are
> > > honoured by the SSVM even after it is rebooted, and even if the
> > > relevant ARP cache entry in the gateway is removed. Of course, this
> is
> > > not the case when the SSVM is destroyed, as the new SSVM will have
> a
> > > different MAC address for every interface.
> > >
> > > It is also interesting noting that in another setup, where we
> > > configured an advanced zone, this problem does not occur. Even the
> > > SSVM is deployed in an adv zone, the network configuration of the
> SSVM
> > > is very similar.
> > >
> > > It would be great if you can provide some advice for debugging this
> > > issue, or share similar experiences.
> > >
> > > Regards,
> > > Salvatore

Mime
View raw message