Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of
 geoff.higginbottom@shapeblue.com designates 216.32.180.187 as permitted
 sender)
Received-SPF: pass (mail88-co1: domain of shapeblue.com designates
 157.56.249.213 as permitted sender) client-ip=157.56.249.213;
 envelope-from=geoff.higginbottom@shapeblue.com;
 helo=AM2PRD0710HT005.eurprd07.prod.outlook.com ;.outlook.com ;
From: Geoff Higginbottom <geoff.higginbottom@shapeblue.com>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: RE: Primary storage failure
Thread-Topic: Primary storage failure
Thread-Index: AQHOeBktVzMgkoKQIUCBRCgWk1edTplTRqmw
Date: Wed, 3 Jul 2013 18:38:06 +0000
Message-ID: 
 <abe2a0bc9cc84f7a81f37f6e7467ea7c@AMXPR07MB070.eurprd07.prod.outlook.com>
References: 
 <CABmj4Rbpo2T-EU3UK5KXN0w-4svQ+goLJREJRpVpi8nqGUhLHw@mail.gmail.com>
In-Reply-To: 
 <CABmj4Rbpo2T-EU3UK5KXN0w-4svQ+goLJREJRpVpi8nqGUhLHw@mail.gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Dean,

I am guessing you are using NFS for your Primary Storage.

This is actually 'by design'.  The logic is that if the storage goes offlin=
e, then all VMs must have also failed, and a 'forced' reboot of the Host 'm=
ight' automatically fix things.

This is great if you only have one Primary Storage, but typically you have =
more than one, so whilst the reboot might fix the failed storage, it will a=
lso kill off all the perfectly good VMs which were still happily running.

The fix for XenServer Hosts is to:

1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting =
out the two entries which have "reboot -f"

2. Identify the PID of the script  - pidof -x xenheartbeat.sh

3. Restart the Script  - kill <pid>

4. Force reconnect Host from the UI,  the script will then re-launch on rec=
onnect

If you running KVM, I'm guessing there is a similar script, but I have not =
tried this yet for anything other than XenSever (it does not apply to ESXi)

Regards

Geoff Higginbottom

D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581

geoff.higginbottom@shapeblue.com


-----Original Message-----
From: Dean Kamali [mailto:dean.kamali@gmail.com]
Sent: 03 July 2013 19:14
To: users@cloudstack.apache.org
Subject: Primary storage failure

Hello everyone

I'm testing failure scenarios, and I have noticed that as soon as the prima=
ry storage gets offline.

cloudstack management server seems to think that the hypervisor is not resp=
onding and it will reboot the node, if you have number of of nodes it will =
eventually reboot all of them. (losing everything  .. fun! )

What if I have multiple primary storage and one of them failed? it will reb=
oot all of my hypervisors? it doesn't seems right to me.

Is there is a way to control this behavior?

it seems that cloud stack management server needs to be a little smarter.
This email and any attachments to it may be confidential and are intended s=
olely for the use of the individual to whom it is addressed. Any views or o=
pinions expressed are solely those of the author and do not necessarily rep=
resent those of Shape Blue Ltd or related companies. If you are not the int=
ended recipient of this email, you must neither take any action based upon =
its contents, nor copy or show it to anyone. Please contact the sender if y=
ou believe you have received this email in error. Shape Blue Ltd is a compa=
ny incorporated in England & Wales. ShapeBlue Services India LLP is operate=
d under license from Shape Blue Ltd. ShapeBlue is a registered trademark.