cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Sonstebo <Dag.Sonst...@shapeblue.com>
Subject Re: Upgrade of Primary storage.
Date Fri, 13 May 2016 10:22:53 GMT
Hi Makrand,

first of all - using the same storage appliance for all purposes (management VMs, primary
and secondary storage) is fairly common, and fully viable as long as you run separate LUNs
/ NFS shares and tie down access accordingly.

With regards to your shutdown / startup steps you're pretty much spot on. You haven't mentioned
if you use NFS or iSCSI but the steps should be more or less the same. 
A couple of suggestions inline below:


A) Shutdown all VMs in following order:- First user VMs, then VRs & then
System VMs.
>>> You probably want to disable the zone first of all to stop any further user access
happening during any shutdown and startup procedures.
>>> With regards to shutting down VMs you can achieve this by putting each primary
storage pool into maintenance - this will shut down all running VMs and system VMs.

B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>>> Once they are in maintenance you will still have to shut them down from XenCenter
or command line.

1) Verify that all hosts are down (Shutdown them one by one) from XEN
Center

C) ssh into Management Server VM.  Backup cloudstack DB save it on JumpBOX
(Use WinSCP)
>>> good practice, make sure you dump both the cloud and cloud_usage databases if
you have both of these present

D) Shutdown Cloudstack Management Server VM and other VMs on management
cluster. This needs to be done by issuing 'shutdown' command from OS level.

E) Shutdown Management cluster XEN hosts.

F) Upgrade the storage.  (Reboot is needed at end.......hence so many
shutdowns)

G) Once upgrade is done start the Machines (Physical and Virtual) in
reveres order than that of shutdown. (Step E to A above) Something like
below

1) Management clsuter XEN hosts
2) Cloudstack management server VM and other management cluster VMs
3) XEN Hypervisor Nodes under cloud.
4) Take XEN hosts out of MM from cloudstack admin GUI.
>>> Take primary storage out of maintenance mode if you followed the suggestion above.
Also re-enable the zone.

5) Start SSVM and console proxy VM (I guess this time cloudstack will
recreate CPVM and SSVM if we      start first user VM.....not sure if this
step will be needed)
>>> Correct, as long as everything is healthy these will be started automatically.

6) Customer VMs and corresponding VRs
H) Verify from and try to deploy test VM.

--

Last time same upgrade happened for one of similar zone and there was issue
of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
that time and hence my manager wants to shutdown all physical hosts this
time (Since all VMs have their disks on this storage box...which will
reboot once or twice during upgrade)
>>> Yes this seems to be a logical precaution, it ensures storage is correctly rescanned
on startup.

--

1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
shutting down all VMs).
>>> yes, in general always make sure CloudStack is aware of what is going on the
underlying hosts, it keeps everything in synch.

2) After I bring UP Management server in step G-1, will it still have host
in MM? or What would it try to do after its up? (Assuming all hosts in
cluster are down at that time.
>>> Yes they will still be in maintenance.




Hope this helps, good luck with your upgrade.


Regards,
Dag Sonstebo
Cloud Architect 
ShapeBlue






On 13/05/2016, 07:58, "Makrand" <makrandsanap@gmail.com> wrote:

>Guys,
>
>Need little help. So in my second week on new job, I am suppose to upgrade
>Nexeta (Primary storage and secondary storage for cloud) OS.
>
>Overview of setup:-
>
>1Zone>>1POD>>1 Cluster>>8 Hosts
>
>Hypervisor:- Citrix Xenserer 6.5 (Free edition)
>
>Storage Primary (Cluster Level):- On nexenta
>
>Secondary Storage:- On same nexenta BOX (which caters primary)
>
>Management Node is VM (Ubuntu 12004 LTS) whos root disk resides again on
>primary storage coming from same nexenta BOX. Seems LUN is different. This
>VM is running on Management cluster (2 XEN 6.5 hosts).
>
>
>Well not sure why and how they kept single storage BOX for all primary and
>secondary. Pretty wiered. It was done by a Vendor or someone long back.
>Thats whole different story.
>
>Any-who, coming to point of this email. I've chalked out plan like below.
>------------------------------------------------------
>A) Shutdown all VMs in following order:- First user VMs, then VRs & then
>System VMs.
>
>B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>
> 1) Verify that all hosts are down (Shutdown them one by one) from XEN
>Center
>
>C) ssh into Management Server VM.  Backup cloudstack DB save it on JumpBOX
>(Use WinSCP)
>
>D) Shutdown Cloudstack Management Server VM and other VMs on management
>cluster. This needs to be done by issuing 'shutdown' command from OS level.
>
>E) Shutdown Management cluster XEN hosts.
>
>F) Upgrade the storage.  (Reboot is needed at end.......hence so many
>shutdowns)
>
>G) Once upgrade is done start the Machines (Physical and Virtual) in
>reveres order than that of shutdown. (Step E to A above) Something like
>below
>
> 1) Management clsuter XEN hosts
> 2) Cloudstack management server VM and other management cluster VMs
> 3) XEN Hypervisor Nodes under cloud.
> 4) Take XEN hosts out of MM from cloudstack admin GUI.
> 5) Start SSVM and console proxy VM (I guess this time cloudstack will
>recreate CPVM and SSVM if we      start first user VM.....not sure if this
>step will be needed)
> 6) Customer VMs and corresponding VRs
>
>H) Verify from and try to deploy test VM.
>------------------------------------------------------
>
>All this downtime is ok. Do you think I am missing anything. Any comments
>on improvising? Should I expect some more glitches? What was your previous
>experience with primary storage upgrade.
>
>Note:-
>Last time same upgrade happened for one of similar zone and there was issue
>of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
>that time and hence my manager wants to shutdown all physical hosts this
>time (Since all VMs have their disks on this storage box...which will
>reboot once or twice during upgrade)
>
>BTW
>
>1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
>shutting down all VMs).
>
>2) After I bring UP Management server in step G-1, will it still have host
>in MM? or What would it try to do after its up? (Assuming all hosts in
>cluster are down at that time.
>
>Thanks for reading.
>
>--
>Best,
>Makrand

Regards,

DagĀ Sonstebo

Dag.Sonstebo@shapeblue.comĀ 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
Mime
View raw message