cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Emrich <martin.emr...@empolis.com>
Subject AW: AW: Packet loss on redundant virtual router since <=4.7.1
Date Fri, 11 Mar 2016 12:34:26 GMT
Hi!

I did the checks:

- If I shut down the backup via Cloudstack, the packet loss disappears.
- VRRP traffic can be seen both on the master (r-1904-VM) and the backup (r-1905-VM).
- The internal gateway IP is only on the master. (eth0)
- The external IP is on both hosts (eth2):

root@r-1904-VM:/var/log# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:2f:2b:00:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.151/23 brd 192.168.101.255 scope global eth0
    inet 192.168.100.1/32 brd 192.168.101.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:01:ce brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.206/16 brd 169.254.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:be:76:00:01:68 brd ff:ff:ff:ff:ff:ff
    inet 10.32.17.103/20 brd 10.32.31.255 scope global eth2


root@r-1905-VM:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:18:ba:00:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.221/23 brd 192.168.101.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:01:f0 brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.240/16 brd 169.254.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:be:76:00:01:68 brd ff:ff:ff:ff:ff:ff
    inet 10.32.17.103/20 brd 10.32.31.255 scope global eth2

So there's something going on with eth2, both are up, and also have the same MAC address.
The keepalived configurations seems to deal only with eth0.

Thanks

Martin 

-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com] 
Gesendet: Montag, 7. März 2016 18:27
An: users@cloudstack.apache.org
Betreff: Re: AW: Packet loss on redundant virtual router since <=4.7.1

The PR mentioned is not related, as it deals with private gateways, which is a vpc feature.


Changing the MAC address will not help as they will still have the same IP address. One interface
is supposed to be down (backup) and one up (master). The first guest tier is used to talk
vrrp so make sure they see each other (check with tcpdump to see if the vrrp announce packets
make it to the other router. If both are master, they are unable to talk to each other and
both assume they are alone. I suspect this is going wrong for you somehow. Please let us know
what tcpdump shows. 

There are several integration tests that deal with this so I'm quite sure it should work.


Regards, Remi 

Sent from my iPhone

> On 07 Mar 2016, at 18:13, Nux! <nux@li.nux.ro> wrote:
> 
> I am not sure on how to apply this, perhaps Wilder or someone else will be able to advise.
> What I would try to do first is to change the MAC addr on the backup VR to something
else and see if that stops the problems.
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Martin Emrich" <martin.emrich@empolis.com>
>> To: users@cloudstack.apache.org
>> Sent: Monday, 7 March, 2016 16:05:02
>> Subject: AW: Packet loss on redundant virtual router since <=4.7.1
> 
>> Hi!
>> 
>> Thanks for the hint! On my test isolated network, currently both 
>> routers have the same configuration on eth2 (the "public" interface), 
>> including the same MAC address, and being up.
>> 
>> So either the VRs do some ARP/ebtables magic to ensure only one 
>> reacts to packets at each given moment, or something is wrong here.
>> 
>> Could that relate to your issue?
>> 
>> I have built ACS RPMs before, what's the easiest way to get a build 
>> with this fix included (just to test whether it applies to us). Note 
>> that we do not use VPC, these are "normal" virtual routers for isolated networks.
>> 
>> Thanks again,
>> 
>> Martin
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Nux! [mailto:nux@li.nux.ro]
>> Gesendet: Montag, 7. März 2016 16:18
>> An: users@cloudstack.apache.org
>> Betreff: Re: Packet loss on redundant virtual router since <=4.7.1
>> 
>> Can you check nex time this happens if the VRs don't have both the 
>> interface up and same mac addr?
>> Could be this issue https://github.com/apache/cloudstack/pull/1413
>> 
>> HTH
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> ----- Original Message -----
>>> From: "Martin Emrich" <martin.emrich@empolis.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Monday, 7 March, 2016 14:00:38
>>> Subject: Packet loss on redundant virtual router since <=4.7.1
>> 
>>> Hello!
>>> 
>>> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
>>> Since then, we have some packet loss with redundant virtual routers.
>>> If I shut down one of the routers (leaving the other one running), 
>>> the problem disappears.
>>> 
>>> I often see messages like "Redundant Virtual Router just switched 
>>> from Unknown to Master".
>>> I also noticed lots of these messages in the messages-logfile on the 
>>> master
>>> router: "Password server failed with error code 1. Restarting it..."
>>> 
>>> What could be wrong?
>>> 
>>> Thanks
>>> 
>>> Martin Emrich
>>> Senior IT Administrator
>>> 
>>> Empolis Information Management GmbH | Europaallee 10 | 67657 
>>> Kaiserslautern | Germany Phone +49 631 68037-71 | Fax +49 631 
>>> 68037-77 
>>> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
>>> 
>>> www.empolis.com<http://www.empolis.com/>
>>> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
>>> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer,
Dr.
>>> Peter Tepassé
>>> 
>>> SMART INFORMATION MANAGEMENT
>>> Empolis-Lösungen befähigen Unternehmen und Organisationen, die 
>>> exponentiell wachsende Menge strukturierter und unstrukturierter 
>>> Daten zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
>>> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische 
>>> Geschäftsprozesse zu optimieren.
>>> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- 
>>> und aufgabengerecht genau die Information, die für sie relevant ist.
>>> Abonnieren Sie unseren
>>> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.2
>>> 3c
>>> tc3r> | Folgen Sie uns auf
>>> Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns 
>>> auf YouTube<http://www.youtube.com/EmpolisSoftware>
>>> [Banner.Experton2016.Signatur]<http://www.empolis.com/>

Mime
View raw message