cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject RE: Help/Advice needed - some traffic don't reach VNET / VM
Date Tue, 10 Oct 2017 06:43:09 GMT
Hi Imran,

Thx for the input, but we are using advanced zone, and guest traffic is
private one...IP is not duplicated.

Thx

On Oct 10, 2017 07:27, "Imran Ahmed" <imran@eaxiom.net> wrote:

> Hi Andrija,
>
> One more thing you can check is that see if there is an IP address
> conflict somewhere.  Please see if the public IP pool assigned to guest VMs
> by CS is somehow overlapping with any IPs which are assigned to physical
> /virtual machines somewhere.
>
> Kind regards,
>
> Imran
>
> -----Original Message-----
> From: Andrija Panic [mailto:andrija.panic@gmail.com]
> Sent: Tuesday, October 10, 2017 2:37 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: Help/Advice needed - some traffic don't reach VNET / VM
>
> Hi guys,
>
> thanks for quick reply:
>
> - VM issue happens on Windows mostly (one customer is of particularly bad
> luck as it seems), but afaik also happens on Linux, and FortiOS (some FW
> stuff, not pure linux) - both are running PV stuff (Windows PV, or CentOS
> 6.5 x64 OS type)
> - we are actually using LACP on switches, and I also disabled/down one bond
> interface on the host - although it makes zero sense, because packer
> already arrived via this bond0, to te bond0.XXX vlan (VTEP), then packet
> also arrived to child vxlan interface (vxlan on top of vlan on top of
> bond...) and then packet also arrived to bridge, but was never passed to
> VNET.
>
> My expectation is that this is purely inside-the-host problem, since packet
> arrives from outside physical network to the host's vxlan/bridge...but not
> to VNET.
> Seems like some qemu issue, but I found zero things using google, that
> looks similar like our issue.
>
> Have no idea...
>
>
>
> On 9 October 2017 at 23:08, Dag Sonstebo <Dag.Sonstebo@shapeblue.com>
> wrote:
>
> > Hi Andrija,
> >
> > Do you use NIC bonds? I have seen this before when using active-active
> > bonds, and as you say it can be very difficult to troubleshoot and the
> > behaviour makes little sense. What can happen is network traffic is load
> > balanced between the two NICs, however the update frequency of the MAC
> > tables between the two switches don’t keep up with the load balanced
> > traffic. In other words a MAC address which used to transmit on
> hypervisor
> > eth0 (attached to your first top of rack switch) of a bond has suddenly
> due
> > to load started transmitting on eth1 (attached to the second of the top
> of
> > rack switches) of the bond, however the physical switch stack still
> thinks
> > the MAC address lives on eth0, hence traffic is dropped until next time
> the
> > switches synch MAC tables.
> >
> > We used to see this a lot in the past on XenServer – the solution being
> > moving to active-passive bond modes, or go up to LACP/802.3ad if your
> > hardware allows for it. The same principle will however also apply on
> > generic linux bonds.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >  S: +44 20 3603 0540  | dag.sonstebo@shapeblue.com |
> > http://www.shapeblue.com <http://www.shapeblue.com/> |
> Twitter:@ShapeBlue
> > <https://twitter.com/#!/shapeblue>
> >
> >
> > On 09/10/2017, 21:52, "Andrija Panic" <andrija.panic@gmail.com> wrote:
> >
> >     Hi guys,
> >
> >     we have occasional but serious problem, that starts happening as it
> > seems
> >     randomly (i.e. NOT under high load)  - not ACS related afaik, purely
> > KVM,
> >     but feedback is really welcomed.
> >
> >     - VM is reachable in general from everywhere, but not reachable from
> >     specific IP address ?!
> >     - VM is NOT under high load, network traffic next to zero, same for
> >     CPU/disk...
> >     - We mitigate this problem by migrating VM away to another host, not
> > much
> >     of a solution...
> >
> >     Description of problem:
> >
> >     We let ping from "problematic" source IP address to the problematic
> > VM, and
> >     we capture traffic on KVM host where the problematic VM lives:
> >
> >     - Tcpdump on VXLAN interface (physical incoming interface on the
> host)
> > - we
> >     see packet fine
> >     - tcpdump on BRIDGE = we see packet fine
> >     - tcpdump on VNET = we DON'T see packet.
> >
> >     In the scenario above, I need to say that :
> >     - we can tcpdump packets from other source IPs on the VNET interface
> > just
> >     fine (as expected), so should also see this problematic source IP's
> > packets
> >     - we can actually ping in oposite direction - from the problematic VM
> > to
> >     the problematic "source" IP
> >
> >     We checked everything possible, from bridge port forwarding, to
> > mac-to-vtep
> >     mapping, to many other things, removed traffic shaping from VNET
> > interface,
> >     no iptables/ebtables, no STP on bridge, remove and rejoin interfaces
> to
> >     bridge, destroy bridge and create manually on the fly,
> >
> >     Problem is really crazy, and I can not explain it - no iptables, no
> >     ebtables for troubleshooting pruposes (on this host) and
> >
> >     We mitigate this problem by migrating VM away to another host, not
> > much of
> >     a solution...
> >
> >     This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1),
> >     Stock kernel 3.16-xx, regular bridge (not OVS)
> >
> >     Anyone else ever heard of such problem - this is not intermittent
> > packet
> >     dropping, but complete blackout/packet drop in some way...
> >
> >     Thanks,
> >
> >     --
> >
> >     Andrija Panić
> >
> >
> >
> > Dag.Sonstebo@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message