cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei ZHOU <ustcweiz...@gmail.com>
Subject Re: Help/Advice needed - some traffic don't reach VNET / VM
Date Fri, 13 Oct 2017 20:28:05 GMT
Hi Andrija,

Good to see your update and know you found the root cause.

-Wei

2017-10-13 22:16 GMT+02:00 Andrija Panic <andrija.panic@gmail.com>:

> Hi all,
>
> I feel obligated to share update, to close the issue:
>
> Nothing to do with kernel/qemu etc.. Seem that hidden Docker NAT/Masquerade
> rules don't play nice with VNET...
>
> Description of the problem as given originally still is valid, but root
> cause is as above...
>
> Apologies for wasting everyone's time and thanks for all the inputs.
>
> Andrija
>
> On 10 October 2017 at 12:18, Wei ZHOU <ustcweizhou@gmail.com> wrote:
>
> > Andrija,
> >
> > We had similar issue before. However, we use advanced zone with security
> > groups, and the issue is because some security groups rules (iptables
> > rules) are not applied by security_group.py successfully.
> > is there any iptables rules on the hypervisors ?
> >
> > -Wei
> >
> > 2017-10-10 11:23 GMT+02:00 Andrija Panic <andrija.panic@gmail.com>:
> >
> > > Hi,
> > >
> > > @Wei, no we are using VXLAN, advanced networking... problem is that
> > packet
> > > not passed from bridge to the VNET - that is "all"...
> > >
> > > @Ivan, we did upgrade few hosts to kernel, 4.4 (made available from
> > Ubuntu
> > > 16.04 to Ubuntu 14.04), but again we there had some issues with FortiOS
> > > (some special OS, not Linux based as I was told), that RDP apps behind
> > this
> > > FW are "slow" (probably laggy), when this FortiGate VM is on new
> > kernel...
> > >
> > > But I'm sure we will move to 4.4, this bug is really driving me
> crazy...
> > :(
> > >
> > > THx
> > >
> > > On 10 October 2017 at 09:52, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com>
> > > wrote:
> > >
> > > > Andrija, I saw it in the past. Problem might be coolnnected with
> kernel
> > > > version and vnet itself. Try to look for it. I don't remember how we
> > > > overcame it in the past...
> > > >
> > > > 10 окт. 2017 г. 8:07 ДП пользователь "Wei ZHOU" <
> ustcweizhou@gmail.com
> > >
> > > > написал:
> > > >
> > > > > Hi Andrija,
> > > > >
> > > > > Are using advanced zone with isolated network or security groups
?
> > > > >
> > > > > -Wei
> > > > >
> > > > >
> > > > > 2017-10-09 22:52 GMT+02:00 Andrija Panic <andrija.panic@gmail.com
> >:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > we have occasional but serious problem, that starts happening
as
> it
> > > > seems
> > > > > > randomly (i.e. NOT under high load)  - not ACS related afaik,
> > purely
> > > > KVM,
> > > > > > but feedback is really welcomed.
> > > > > >
> > > > > > - VM is reachable in general from everywhere, but not reachable
> > from
> > > > > > specific IP address ?!
> > > > > > - VM is NOT under high load, network traffic next to zero, same
> for
> > > > > > CPU/disk...
> > > > > > - We mitigate this problem by migrating VM away to another host,
> > not
> > > > much
> > > > > > of a solution...
> > > > > >
> > > > > > Description of problem:
> > > > > >
> > > > > > We let ping from "problematic" source IP address to the
> problematic
> > > VM,
> > > > > and
> > > > > > we capture traffic on KVM host where the problematic VM lives:
> > > > > >
> > > > > > - Tcpdump on VXLAN interface (physical incoming interface on
the
> > > host)
> > > > -
> > > > > we
> > > > > > see packet fine
> > > > > > - tcpdump on BRIDGE = we see packet fine
> > > > > > - tcpdump on VNET = we DON'T see packet.
> > > > > >
> > > > > > In the scenario above, I need to say that :
> > > > > > - we can tcpdump packets from other source IPs on the VNET
> > interface
> > > > just
> > > > > > fine (as expected), so should also see this problematic source
> IP's
> > > > > packets
> > > > > > - we can actually ping in oposite direction - from the
> problematic
> > VM
> > > > to
> > > > > > the problematic "source" IP
> > > > > >
> > > > > > We checked everything possible, from bridge port forwarding,
to
> > > > > mac-to-vtep
> > > > > > mapping, to many other things, removed traffic shaping from
VNET
> > > > > interface,
> > > > > > no iptables/ebtables, no STP on bridge, remove and rejoin
> > interfaces
> > > to
> > > > > > bridge, destroy bridge and create manually on the fly,
> > > > > >
> > > > > > Problem is really crazy, and I can not explain it - no iptables,
> no
> > > > > > ebtables for troubleshooting pruposes (on this host) and
> > > > > >
> > > > > > We mitigate this problem by migrating VM away to another host,
> not
> > > much
> > > > > of
> > > > > > a solution...
> > > > > >
> > > > > > This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1),
> > > > > > Stock kernel 3.16-xx, regular bridge (not OVS)
> > > > > >
> > > > > > Anyone else ever heard of such problem - this is not intermittent
> > > > packet
> > > > > > dropping, but complete blackout/packet drop in some way...
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Andrija Panić
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
>
>
>
> --
>
> Andrija Panić
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message