cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Sonstebo <Dag.Sonst...@shapeblue.com>
Subject Re: AW: 2node HA ACS Cluster with DRBD
Date Fri, 25 Nov 2016 09:32:42 GMT
Hi Jeoroen,

Sorry I missed you yesterday –  meant to catch up after the user group but had to run to
catch my flight.

I think what you describe will work – however I have my doubts it will fail over gracefully.
For pacemaker to fail over cleanly the failover has to be perfectly synched – i.e. all packets
have to be written in both primary storage pools, traffic ideally quiesced – then pacemaker
can move the NFS or iSCSI endpoint. If you are even a byte out you could end up with data
corruption – and even if this does work I have my doubts the VMs would stay online afterwards.

Anyway – as the proverb goes, the proof is in the pudding – so I can only suggest you
test this out. Very interested in the result though – so please let us know how you get
on (if it works it would be a good talk for the next user group ☺ ).

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

From: Jeroen Keerl <jeroen.keerl@keerl-it.com>
Reply-To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>, "jeroen.keerl@keerl-it.com"
<jeroen.keerl@keerl-it.com>
Date: Wednesday, 23 November 2016 at 22:43
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: AW: 2node HA ACS Cluster with DRBD

Hi Dag, Erik,

thanks for your input so far.
What I am aiming for is a "HyperConverged" infrastructure, if possible with just two servers.

The reason why I didn't look into Ceph any further, is that they explicitly state that they'll
need 3 hosts.
Apart from that, the seems to be quite a lot of resource needs to get Ceph up & running.

DRBD and GlusterFS look like they're not that heavy on load.
GlusterFS has moved away from 2 hosts only as well, and it seems less flexible when it comes
to expansion, if I recall correctly.

Hence: DRBD, which runs in Master-Slave or Dual Master mode.
Together with Pacemaker and NFS or iSCSI software, this could work, albeit -after overthinking
it all- probably in a master-slave mode, since the shared / clustered IP address can only
be available on one of two nodes.

As written before: HA-Lizard does all this out of the box, including HA - if needed, and fairly
well too.

Since I'll be hopping over to visit the CS User Group tomorrow, I'll have no time to look
into this any further until Tuesday.
(Dag, will I have to chance to see you there as well?)

Cheers
JK

Dag.Sonstebo@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Ursprüngliche Nachricht-----
Von: Dag Sonstebo [mailto:Dag.Sonstebo@shapeblue.com<mailto:Dag.Sonstebo@shapeblue.com>]
Gesendet: Mittwoch, 23. November 2016 10:35
An: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Betreff: Re: 2node HA ACS Cluster with DRBD

Hi Jeroen,

My twopence worth:

First of all – I guess your plan is to run two nodes – each with CloudStack management,
MySQL (master-slave), KVM and storage?

This all depends on your user case. As a bit of an experiment or as a small scale lab I think
this may work – but I would be very reluctant to rely on this for a production workload.
I think you will potentially have stability and integrity issues at the storage level in a
HA failover scenario, on top of this I don’t think this will scale well. You may also end
up with considerable storage overhead depending on number of nodes + technology used. With
two nodes you immediately only have 50% max space utilization.

Putting all of that aside I think it could work, I’ve played with similar ideas in the past
(without actually spending enough time to get it working). I think you could get around the
heartbeating / split brain situations relatively easily. The CloudStack and MySQL installs
as well as KVM should work OK, but your challenge will be storage, which both has to work
in the synchronized setup you want + very importantly fail over gracefully. I guess you would
probably look at Ceph - if Wido or any of the other Ceph users read this they are probably
better placed to advise.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

From: Jeroen Keerl <jeroen.keerl@keerl-it.com<mailto:jeroen.keerl@keerl-it.com>>
Reply-To: "users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>" <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>,
"jeroen.keerl@keerl-it.com<mailto:jeroen.keerl@keerl-it.com>" <jeroen.keerl@keerl-it.com<mailto:jeroen.keerl@keerl-it.com>>
Date: Wednesday, 23 November 2016 at 00:05
To: "users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>" <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
Subject: 2node HA ACS Cluster with DRBD

All,

I’ve been pondering a bit about a possible small-scale, HA setup:

Hardware
2 Nodes, each with 2 1Gbe and 2 10Gbe NICs Both have hardware RAID controllers, redundant
PS, Array controller with BBWC and hot plug SAS drives.

If we’d “combine” the quick setup for CentOS6 ( http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/qig.html
) with the “additional management server” part ( http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/management-server/index.html#additional-management-servers
) then we’d have 2 management servers.
To make the local storage of both nodes “HA”, one could use DRBD in a “primary-primary”
setup, so VM migration is possible as well. (I am not yet clear on how to present this storage
to the nodes)

To avoid split brain-situations, I see two possible strategi
es:

1)       Direct cabling of the 10Gbe controllers => Loss of connection can only mean either
one host is dead, or at least its NIC has died.

2)       Usage of IPMI / Fencing

Did anybody gather experience in a setup like that?
Does anybody have any thoughts on this – improvements, comments, doubts: Hit me!

Cheers,
JK


·         I came to this “setup”, after a few issues with Xen Server 6.5 combined with
“HA-Lizard”, which actually uses DRBD in combination with TGTD iSCSI. On the one side,
I’ve had quite a few stability issues with Xen 6.5 and on the other side: The CS MS needs
to be outside of this cluster … but still needs to be installed on a redundant / HA piece
of kit.




Jeroen Keerl

Keerl IT Services GmbH
Birkenstraße 1b . 21521 Aumühle

+49 177 6320 317

www.keerl-it.com<http://www.keerl-it.com><http://www.keerl-it.com/>
info@keerl-it.com<mailto:info@keerl-it.com><mailto:info@keerl-it.com<mailto:info@keerl-it.com>>

Geschäftsführer. Jacobus J. Keerl
Registergericht Lubeck. HRB-Nr. 14511

Unsere Allgemeine Geschäftsbedingungen finden Sie hier.<http://www.keerl-it.com/AGB.pdf>

[cid:d3544f14.06fb964e.PNG.90ac9d6c]



Dag.Sonstebo@shapeblue.com<mailto:Dag.Sonstebo@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue





Jeroen Keerl

Keerl IT Services GmbH
Birkenstraße 1b . 21521 Aumühle

+49 177 6320 317

www.keerl-it.com<http://www.keerl-it.com/>
info@keerl-it.com<mailto:info@keerl-it.com>

Geschäftsführer. Jacobus J. Keerl
Registergericht Lubeck. HRB-Nr. 14511

Unsere Allgemeine Geschäftsbedingungen finden Sie hier.<http://www.keerl-it.com/AGB.pdf>

[cid:d3544f14.06fb964e.PNG.d5855133]


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message