cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Disable HA temporary ?
Date Mon, 16 Feb 2015 15:43:59 GMT
Seems to me, that I'm about to issue something similar to:   update
cloud.vm_instance set ha = 0 where ha =1...

Now seriously, wondering, per the manual - if you define HA host tag on the
global config level, and then have NO hosts with that tag - MGMT will not
be able to start VMs on other hosts, since there are no hosts that are
dedicated got HA destination ?

Does this makes sense ? I guess the VMs will be just marked as Stopped in
the GUI/databse, but unable to start them...
Stupid proposal, but... ?

On 16 February 2015 at 16:22, Logan Barfield <lbarfield@tqhosting.com>
wrote:

> Some sort of fencing independent of the management server is
> definitely needed.  HA in general (particularly on KVM) is all kinds
> of unpredictable/buggy right now.
>
> I like the idea of having a switch that an admin can flip to stop HA.
> In fact I think a better job control system in general (e.g., being
> able to stop/restart/manually start tasks) would be awesome, if it's
> feasible.
>
> Thank You,
>
> Logan Barfield
> Tranquil Hosting
>
>
> On Mon, Feb 16, 2015 at 10:05 AM, Wido den Hollander <wido@widodh.nl>
> wrote:
> >
> >
> > On 16-02-15 13:16, Andrei Mikhailovsky wrote:
> >> I had similar issues at least two or thee times. The host agent would
> disconnect from the management server. The agent would not connect back to
> the management server without manual intervention, however, it would
> happily continue running the vms. The management server would initiate the
> HA and fire up vms, which are already running on the disconnected host. I
> ended up with a handful of vms and virtual routers being ran on two
> hypervisors, thus corrupting the disk and having all sorts of issues ((( .
> >>
> >> I think there has to be a better way of dealing with this case. At
> least on an image level. Perhaps a host should keep some sort of lock file
> or a file for every image where it would record a time stamp. Something
> like:
> >>
> >> f5ffa8b0-d852-41c8-a386-6efb8241e2e7 and
> >> f5ffa8b0-d852-41c8-a386-6efb8241e2e7-timestamp
> >>
> >> Thus, the f5ffa8b0-d852-41c8-a386-6efb8241e2e7 is the name of the disk
> image and f5ffa8b0-d852-41c8-a386-6efb8241e2e7-timestamp is the image's
> time stamp.
> >>
> >> The hypervisor should record the time stamp in this file while the vm
> is running. Let's say every 5-10 seconds. If the timestamp is old, we can
> assume that the volume is no longer used by the hypervisor.
> >>
> >> When a vm is started, the timestamp file should be checked and if the
> timestamp is recent, the vm should not start, otherwise, the vm should
> start and the timestamp file should be regularly updated.
> >>
> >> I am sure there are better ways of doing this, but at least this method
> should not allow two vms running on different hosts to use the same volume
> and corrupt the data.
> >>
> >> In ceph, as far as I remember, a new feature is being developed to
> provide a locking mechanism of an rbd image. Not sure if this will do the
> job?
> >>
> >
> > Something like this is still on my wishlist for Ceph/RBD, something like
> > you propose.
> >
> > For NFS we currently have this in place, but for Ceph/RBD we don't. It's
> > a matter of code in the Agent and the investigators inside the
> > Management Server which decide if HA should kick in.
> >
> > Wido
> >
> >> Andrei
> >>
> >> ----- Original Message -----
> >>
> >>> From: "Wido den Hollander" <wido@widodh.nl>
> >>> To: dev@cloudstack.apache.org
> >>> Sent: Monday, 16 February, 2015 11:32:13 AM
> >>> Subject: Re: Disable HA temporary ?
> >>
> >>> On 16-02-15 11:00, Andrija Panic wrote:
> >>>> Hi team,
> >>>>
> >>>> I just had funny behaviour few days ago - one of my hosts was under
> >>>> heavy
> >>>> load (some disk/network load) and it went disconnected from MGMT
> >>>> server.
> >>>>
> >>>> Then MGMT server stared doing HA thing, but without being able to
> >>>> make sure
> >>>> that the VMs on the disconnected hosts are really shutdown (and
> >>>> they were
> >>>> NOT).
> >>>>
> >>>> So MGMT started again some VMs on other hosts, thus resulting in
> >>>> having 2
> >>>> copies of the same VM, using shared strage - so corruption happened
> >>>> on the
> >>>> disk.
> >>>>
> >>>> Is there a way to temporary disable HA feature on global level, or
> >>>> anything
> >>>> similar ?
> >>
> >>> Not that I'm aware of, but this is something I also ran in to a
> >>> couple
> >>> of times.
> >>
> >>> It would indeed be nice if there could be a way to stop the HA
> >>> process
> >>> completely as an Admin.
> >>
> >>> Wido
> >>
> >>>> Thanks
> >>>>
> >>
>



-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message