Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+FPaeYjBpuDNO8EZL2jW0qik7LmWy8b8k=4DgFj9ntT4-9iRg@mail.gmail.com>
References: <CAKLH8EbV6OFd7Cx4Wi5-n0L2Ly7bki_Phcd3CgqKLLsHR8u=BA@mail.gmail.com>
 <CA+FPaeYjBpuDNO8EZL2jW0qik7LmWy8b8k=4DgFj9ntT4-9iRg@mail.gmail.com>
From: =?UTF-8?Q?Rafael_Weing=C3=A4rtner?= <rafaelweingartner@gmail.com>
Date: Mon, 18 Dec 2017 07:20:25 -0200
Message-ID: <CAG97rac6ZAk0Kjhv6fwOvtRK2DaTBcy9ZVfJwV1CL_OCEswKnw@mail.gmail.com>
Subject: Re: [Discuss] Management cluster / Zookeeper holding locks
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Cc: dev <dev@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary="001a1138e9b44a961b056099dc0b"
archived-at: Mon, 18 Dec 2017 09:20:32 -0000

--001a1138e9b44a961b056099dc0b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

How hard is it to configure Zookeeper and get everything up and running?
BTW: what zookeeper would be managing? CloudStack management servers or
MySQL nodes?

On Mon, Dec 18, 2017 at 7:13 AM, Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com=
>
wrote:

> Hello, Marc-Aurele, I strongly believe that all mysql locks should be
> removed in favour of truly DLM solution like Zookeeper. The performance o=
f
> 3node ZK ensemble should be enough to hold up to 1000-2000 locks per seco=
nd
> and it helps to move to truly clustered MySQL like galera without single
> master server.
>
> 2017-12-18 15:33 GMT+07:00 Marc-Aur=C3=A8le Brothier <marco@exoscale.ch>:
>
> > Hi everyone,
> >
> > I was wondering how many of you are running CloudStack with a cluster o=
f
> > management servers. I would think most of you, but it would be nice to
> hear
> > everyone voices. And do you get hosts going over their capacity limits?
> >
> > We discovered that during the VM allocation, if you get a lot of parall=
el
> > requests to create new VMs, most notably with large profiles, the
> capacity
> > increase is done too far after the host capacity checks and results in
> > hosts going over their capacity limits. To detail the steps: the
> deployment
> > planner checks for cluster/host capacity and pick up one deployment pla=
n
> > (zone, cluster, host). The plan is stored in the database under a VMwor=
k
> > job and another thread picks that entry and starts the deployment,
> > increasing the host capacity and sending the commands. Here there's a
> time
> > gap between the host being picked up and the capacity increase for that
> > host of a couple of seconds, which is well enough to go over the capaci=
ty
> > on one or more hosts. A few VMwork job can be added in the DB queue
> > targeting the same host before one gets picked up.
> >
> > To fix this issue, we're using Zookeeper to act as the multi JVM lock
> > manager thanks to their curator library (
> > https://curator.apache.org/curator-recipes/shared-lock.html). We also
> > changed the time when the capacity is increased, which occurs now prett=
y
> > much after the deployment plan is found and inside the zookeeper lock.
> This
> > ensure we don't go over the capacity of any host, and it has been prove=
n
> > efficient since a month in our management server cluster.
> >
> > This adds another potential requirement which should be discuss before
> > proposing a PR. Today the code works seamlessly without ZK too, to ensu=
re
> > it's not a hard requirement, for example in a lab.
> >
> > Comments?
> >
> > Kind regards,
> > Marc-Aur=C3=A8le
> >
>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>


--=20
Rafael Weing=C3=A4rtner

--001a1138e9b44a961b056099dc0b--