cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: [DISCUSS] Management server (pre-)shutdown to avoid killing jobs
Date Tue, 19 Dec 2017 10:04:54 GMT
This is the best design, Marc. I also find quite …..  seeing all of those
thread being managed manually in ACS. I would only look at other
technologies such as Spring-integration or AKKA to create the structure for
an async messaging system across nodes/JVMs (I find Kafka too damn
complicated to install and maintain). There is a big problem though. In a
system that we design and code from scratch is quite easy to create a
project, plan and implement a new and resilient design like this. However,
in legacy code such as ACS, how can we proceed to this type of solution?

We would need a plan to extract/move bit by bit to this new design. For
instance, starting to recreate system VMs or host’s agents (these are the
most isolated piece of code I could think of). It’s time to think about ACS
5.0, this might be our chance…

On Tue, Dec 19, 2017 at 5:29 AM, Marc-Aurèle Brothier <marco@exoscale.ch>
wrote:

> It's definitively a great direction to take and much more robust. ZK would
> be great fit to monitor the state of management servers and agent with the
> help of the ephemeral nodes. On the other side, it's not encouraged to use
> it as a messaging queue, and kafka would be a much better fit for that
> purpose, having partitions/topics. Doing a quick overview of the
> architecture I would see ZK used a inter-JVM lock, holding mgmt & agent
> status nodes, along with their capacities using a direct connection from
> each of them to ZK. Kafka would be use to exchange the current command
> messages between management servers, management server & agents. With those
> 2 kind of brokers in the middle it will make the system super resilient.
> For example, if a management sends a command to stop a VM on a host, but
> that host's agent is stopping to perform an upgrade, when it connects back
> to the kafka topic, its "stop" message would still be there if it didn't
> expired and the command could be process. Of course it would have taken
> more time, but still, it would not return an error message. This would
> remove the need to create and manage threads in the management server to
> handle all the async tasks & checks and move it to an event driven
> approach. At the same time it adds 2 dependencies that require setup &
> configuration, moving out of the goal to have an easy, almost all-included,
> installable solution... Trade-offs to be discussed.
>
> On Mon, Dec 18, 2017 at 8:06 PM, ilya musayev <
> ilya.mailing.lists@gmail.com>
> wrote:
>
> > I very much agree with Paul, we should consider moving into resilient
> model
> > with least dependence I.e ha-proxy..
> >
> > Send a notification to partner MS to take over the job management would
> be
> > ideal.
> >
> > On Mon, Dec 18, 2017 at 9:28 AM Paul Angus <paul.angus@shapeblue.com>
> > wrote:
> >
> > > Hi Marc-Aurèle,
> > >
> > > Personally, my utopia would be to be able to pass async jobs between
> > mgmt.
> > > servers.
> > > So rather than waiting in indeterminate time for a snapshot to
> complete,
> > > monitoring the job is passed to another management server.
> > >
> > > I would LOVE that something like Zookeeper monitored the state of the
> > > mgmt. servers, so that 'other' management servers could take over the
> > async
> > > jobs in the (unlikely) event that a management server becomes
> > unavailable.
> > >
> > >
> > >
> > > Kind regards,
> > >
> > > Paul Angus
> > >
> > > paul.angus@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Marc-Aurèle Brothier [mailto:marco@exoscale.ch]
> > > Sent: 18 December 2017 13:56
> > > To: dev@cloudstack.apache.org
> > > Subject: [DISCUSS] Management server (pre-)shutdown to avoid killing
> jobs
> > >
> > > Hi everyone,
> > >
> > > Another point, another thread. Currently when shutting down a
> management
> > > server, despite all the "stop()" method not being called as far as I
> > know,
> > > the server could be in the middle of processing an async job task. It
> > will
> > > lead to a failed job since the response won't be delivered to the
> correct
> > > management server even though the job might have succeed on the agent.
> To
> > > overcome this limitation due to our weekly production upgrades, we
> added
> > a
> > > pre-shutdown mechanism which works along side HA-proxy. The management
> > > server keeps a eye onto a file "lb-agent" in which some keywords can be
> > > written following the HA proxy guide (
> > > https://cbonte.github.io/haproxy-dconv/1.9/
> configuration.html#5.2-agent-
> > check
> > > ).
> > > When it finds "maint", "stopped" or "drain", it stops those threads:
> > >  - AsyncJobManager._heartbeatScheduler: responsible to fetch and start
> > > execution of AsyncJobs
> > >  - AlertManagerImpl._timer: responsible to send capacity check commands
> > >  - StatsCollector._executor: responsible to schedule stats command
> > >
> > > Then the management server stops most of its scheduled tasks. The
> correct
> > > thing to do before shutting down the server would be to send
> > > "rebalance/reconnect" commands to all agents connected on that
> management
> > > server to ensure that commands won't go through this server at all.
> > >
> > > Here, HA-proxy is responsible to stop sending API requests to the
> > > corresponding server with the help of this local agent check.
> > >
> > > In case you want to cancel the maintenance shutdown, you could write
> > > "up/ready" in the file and the different schedulers will be restarted.
> > >
> > > This is really more a change for operation around CS for people doing
> > live
> > > upgrade on a regular basis, so I'm unsure if the community would want
> > such
> > > a change in the code base. It goes a bit in the opposite direction of
> the
> > > change for removing the need of HA-proxy
> > > https://github.com/apache/cloudstack/pull/2309
> > >
> > > If there is enough positive feedback for such a change, I will port
> them
> > > to match with the upstream branch in a PR.
> > >
> > > Kind regards,
> > > Marc-Aurèle
> > >
> >
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message