cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lennert den Teuling (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-3954) HA with Security Groups and ping disabled will cause split-brian
Date Tue, 06 Aug 2013 07:56:51 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730494#comment-13730494
] 

Lennert den Teuling commented on CLOUDSTACK-3954:
-------------------------------------------------

Hi Koushik,

It's true that it only happends when the agent crashes or you kill it. If you shut it down
properly there is no HA cause another disconnect status is send to the manager. This is right,
cause you should be able to restart the agent for example without triggering HA. 

I think the main concern here is that besides the agent there is no second method to verify
if a VM is running. So when the agent crashes, the heartbeat for the fencer will also stop
working because it is the same process. I don't see the VM ping check as a valid second way
to determine if a VM is running cause there could be nummerous reason why a VM is not pingable
(SG, Firewall on the VM etc).

Currently the agent is quite stable, but there could be a lot of (future) reasons why it could
crash. If it crashes the split-brain will occur. 

So I think we should replace the VM ping check with something more reliable. For example my
colleague is currently working on a simple webservice which runs as a seperate process on
the host and can tell the management service if VMs are running on a host even when the agent
is not running. This way you always have 2 ways of knowing if a VM is alive, and is way more
reliable than a ping. 

Do you see this as a solution? 



                
> HA with Security Groups and ping disabled will cause split-brian
> ----------------------------------------------------------------
>
>                 Key: CLOUDSTACK-3954
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3954
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: KVM
>    Affects Versions: 4.1.0
>         Environment: Tested this with CS 4.1 on Ubuntu, but will probably exist in other
versions
>            Reporter: Lennert den Teuling
>            Priority: Critical
>             Fix For: 4.2.0
>
>
> We found out that when running CS 4.1 on KVM with Security Groups enabled + ping disabled
(default) will cause a split-brain when agent crashes. 
> How to reproduce:
> 1. Setup a Basic Zone with SG enabled
> 2. Create one or multiple  HA-enabled VMs with a security group which does not allow
ping (by default). 
> 3. Kill the agent on one of the hosts
> When you do this, the HA component on the management server will restart all VMs on another
node, even when they are running and the VM host is still pingable. This will likely corrupt
all VMs on the host where the agent was stopped/killed. 
> We had some issues with libvirt causing the agent to disconnect. Luckily some VMs allowed
ping so nothing bad happened.  
> Temporary fix:
> Ensure at least one of the running VMs on each hosts allows ping, so the HA manager will
be able to ping it and will not HA the host. 
> I'm not sure yet why this happens, but wanted to file this bug so people can take necessary
preparations. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message