cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Weller <swel...@ena.com.INVALID>
Subject Re: Help/Advice needed - some traffic don't reach VNET / VM
Date Mon, 09 Oct 2017 21:05:37 GMT
Andrija,


What is the guest OS for this VM, or does this issue not discriminate?

- Si
________________________________
From: Andrija Panic <andrija.panic@gmail.com>
Sent: Monday, October 9, 2017 3:52 PM
To: dev@cloudstack.apache.org; users@cloudstack.apache.org
Subject: Help/Advice needed - some traffic don't reach VNET / VM

Hi guys,

we have occasional but serious problem, that starts happening as it seems
randomly (i.e. NOT under high load)  - not ACS related afaik, purely KVM,
but feedback is really welcomed.

- VM is reachable in general from everywhere, but not reachable from
specific IP address ?!
- VM is NOT under high load, network traffic next to zero, same for
CPU/disk...
- We mitigate this problem by migrating VM away to another host, not much
of a solution...

Description of problem:

We let ping from "problematic" source IP address to the problematic VM, and
we capture traffic on KVM host where the problematic VM lives:

- Tcpdump on VXLAN interface (physical incoming interface on the host) - we
see packet fine
- tcpdump on BRIDGE = we see packet fine
- tcpdump on VNET = we DON'T see packet.

In the scenario above, I need to say that :
- we can tcpdump packets from other source IPs on the VNET interface just
fine (as expected), so should also see this problematic source IP's packets
- we can actually ping in oposite direction - from the problematic VM to
the problematic "source" IP

We checked everything possible, from bridge port forwarding, to mac-to-vtep
mapping, to many other things, removed traffic shaping from VNET interface,
no iptables/ebtables, no STP on bridge, remove and rejoin interfaces to
bridge, destroy bridge and create manually on the fly,

Problem is really crazy, and I can not explain it - no iptables, no
ebtables for troubleshooting pruposes (on this host) and

We mitigate this problem by migrating VM away to another host, not much of
a solution...

This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1),
Stock kernel 3.16-xx, regular bridge (not OVS)

Anyone else ever heard of such problem - this is not intermittent packet
dropping, but complete blackout/packet drop in some way...

Thanks,

--

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message