cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Gipson <tgip...@ena.com.INVALID>
Subject Re: Question concerning Virtual Routers and problems during failover
Date Wed, 13 Sep 2017 20:17:41 GMT
I just opened a JIRA issue  https://issues.apache.org/jira/browse/CLOUDSTACK-10074 and added
my IPtables files as well as the management logs and the logs from the routers.  I started
the manual failover at around 14:40 so that should help anyone wanting to look at the logs.

Thanks!

Tim Gipson
Systems Engineer
Direct: 615-312-6157
Mobile: 615-585-3652

 <https://www.ena.com/>
 

On 9/13/17, 10:18 AM, "Tim Gipson" <tgipson@ena.com.INVALID> wrote:

    Sure, I’ll need to recreate a failure scenario so I can capture all that data for you.
 I’ll post it here as soon as I’ve got it.
    
    Thanks!
    
    Tim Gipson
    Systems Engineer
    Direct: 615-312-6157
    Mobile: 615-585-3652
    
     <https://www.ena.com/>
     
    
    On 9/12/17, 10:53 PM, "Nitin Kumar Maharana" <nitinkumar.maharana@accelerite.com>
wrote:
    
        Hi Tim,
        
        Can you please attach both VR’s cloud.log(present in VR path /var/log/cloud.log)
as well as management server log of the failure case.
        Which will help us finding out the exact cause of the failure.
        
        
        Thanks,
        Nitin
        On 13-Sep-2017, at 12:42 AM, Tim Gipson <tgipson@ena.com.INVALID<mailto:tgipson@ena.com.invalid>>
wrote:
        
        Hey all,
        
        I’ve found what I think could be a possible issue with the redundant VPC router
pairs in Clousdstack.  The issue was first noticed when routers were failing over from master
to backup.  When the backup router became master, everything continued to work properly and
traffic flowed as normal.  However, when it failed from the new master back to the original
master the virtual router stopped allowing traffic through any network interfaces and any
failover after that resulted in virtual routers that were not passing traffic.
        
        I can reproduce this behavior by doing a manual failover (logging in and issuing a
reboot command on the router) from master to backup and then back to the original master.
 From what I can tell, the iptables rules on the router are somehow modified during the failover
(or a manual reboot) in such a way as to make them completely nonfunctional.  I did a side-by-side
comparison of the iptables rules before and after a failover (or a manual reboot) and there
are definite differences.  Sometimes rules are changed, sometimes they are duplicated, and
I’ve even found that some rules are missing completely out of iptables.
        
        We are running in a CentOS 7 environment and using KVM as our hypervisor.  Our CS
version is 4.8 with standard images for the VRs.  As mentioned previously, our VRs are in
redundant pairs for VPCs.
        
        I’ve attached two iptables outputs, one from a working router and one from a broken
router after failover.
        
        Any help or direction you could provide to help me further identify why this is happening
would be appreciated.
        
        Thanks!
        
        Tim Gipson
        <https://www.ena.com/>
        
        
        
        <iptables_broken.txt><iptables_working.txt>
        
        DISCLAIMER
        ==========
        This e-mail may contain privileged and confidential information which is the property
of Accelerite, a Persistent Systems business. It is intended only for the use of the individual
or entity to which it is addressed. If you are not the intended recipient, you are not authorized
to read, retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent
Systems business does not accept any liability for virus infected mails.
        
    
    

Mime
View raw message