Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+FPaea1_QK-C304uGRZpXx8utPZed=4COyfdze5EuS2DxHfXw@mail.gmail.com>
References: <CAKLH8EbV6OFd7Cx4Wi5-n0L2Ly7bki_Phcd3CgqKLLsHR8u=BA@mail.gmail.com>
 <CA+FPaeYjBpuDNO8EZL2jW0qik7LmWy8b8k=4DgFj9ntT4-9iRg@mail.gmail.com>
 <CAG97rac6ZAk0Kjhv6fwOvtRK2DaTBcy9ZVfJwV1CL_OCEswKnw@mail.gmail.com> <CA+FPaea1_QK-C304uGRZpXx8utPZed=4COyfdze5EuS2DxHfXw@mail.gmail.com>
From: =?UTF-8?Q?Marc=2DAur=C3=A8le_Brothier?= <marco@exoscale.ch>
Date: Mon, 18 Dec 2017 10:47:51 +0100
Message-ID: <CAKLH8Eah0D_Vmb2B8V5NsD1juTkt+0CyRrQjKew0kq1gunA6hQ@mail.gmail.com>
Subject: Re: [Discuss] Management cluster / Zookeeper holding locks
To: dev@cloudstack.apache.org
Cc: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary="001a114710a46e3a4005609a3ea7"
archived-at: Mon, 18 Dec 2017 09:48:04 -0000

--001a114710a46e3a4005609a3ea7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

We added ZK lock for fix this issue but we will remove all current locks in
ZK in favor of ZK one. The ZK lock is already encapsulated in a project
with an interface, but more work should be done to have a proper interface
for locks which could be implemented with the "tool" you want, either a DB
lock for simplicity, or ZK for more advanced scenarios.

@Daan you will need to add the ZK libraries in CS and have a running ZK
server somewhere. The configuration value is read from the
server.properties. If the line is empty, the ZK client is not created and
any lock request will immediately return (not holding any lock).

@Rafael: ZK is pretty easy to setup and have running, as long as you don't
put too much data in it. Regarding our scenario here, with only locks, it's
easy. ZK would be only the gatekeeper to locks in the code, ensuring that
multi JVM can request a true lock.
For the code point of view, you're opening a connection to a ZK node (any
of a cluster) and you create a new InterProcessSemaphoreMutex which handles
the locking mechanism.

On Mon, Dec 18, 2017 at 10:24 AM, Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.co=
m
> wrote:

> Rafael,
>
> - It's easy to configure and run ZK either in single node or cluster
> - zookeeper should replace mysql locking mechanism used inside ACS code
> (places where ACS locks tables or rows).
>
> I don't think from the other size, that moving from MySQL locks to ZK loc=
ks
> is easy and light and (even implemetable) way.
>
> 2017-12-18 16:20 GMT+07:00 Rafael Weing=C3=A4rtner <rafaelweingartner@gma=
il.com
> >:
>
> > How hard is it to configure Zookeeper and get everything up and running=
?
> > BTW: what zookeeper would be managing? CloudStack management servers or
> > MySQL nodes?
> >
> > On Mon, Dec 18, 2017 at 7:13 AM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com>
> > wrote:
> >
> > > Hello, Marc-Aurele, I strongly believe that all mysql locks should be
> > > removed in favour of truly DLM solution like Zookeeper. The performan=
ce
> > of
> > > 3node ZK ensemble should be enough to hold up to 1000-2000 locks per
> > second
> > > and it helps to move to truly clustered MySQL like galera without
> single
> > > master server.
> > >
> > > 2017-12-18 15:33 GMT+07:00 Marc-Aur=C3=A8le Brothier <marco@exoscale.=
ch>:
> > >
> > > > Hi everyone,
> > > >
> > > > I was wondering how many of you are running CloudStack with a clust=
er
> > of
> > > > management servers. I would think most of you, but it would be nice
> to
> > > hear
> > > > everyone voices. And do you get hosts going over their capacity
> limits?
> > > >
> > > > We discovered that during the VM allocation, if you get a lot of
> > parallel
> > > > requests to create new VMs, most notably with large profiles, the
> > > capacity
> > > > increase is done too far after the host capacity checks and results
> in
> > > > hosts going over their capacity limits. To detail the steps: the
> > > deployment
> > > > planner checks for cluster/host capacity and pick up one deployment
> > plan
> > > > (zone, cluster, host). The plan is stored in the database under a
> > VMwork
> > > > job and another thread picks that entry and starts the deployment,
> > > > increasing the host capacity and sending the commands. Here there's=
 a
> > > time
> > > > gap between the host being picked up and the capacity increase for
> that
> > > > host of a couple of seconds, which is well enough to go over the
> > capacity
> > > > on one or more hosts. A few VMwork job can be added in the DB queue
> > > > targeting the same host before one gets picked up.
> > > >
> > > > To fix this issue, we're using Zookeeper to act as the multi JVM lo=
ck
> > > > manager thanks to their curator library (
> > > > https://curator.apache.org/curator-recipes/shared-lock.html). We
> also
> > > > changed the time when the capacity is increased, which occurs now
> > pretty
> > > > much after the deployment plan is found and inside the zookeeper
> lock.
> > > This
> > > > ensure we don't go over the capacity of any host, and it has been
> > proven
> > > > efficient since a month in our management server cluster.
> > > >
> > > > This adds another potential requirement which should be discuss
> before
> > > > proposing a PR. Today the code works seamlessly without ZK too, to
> > ensure
> > > > it's not a hard requirement, for example in a lab.
> > > >
> > > > Comments?
> > > >
> > > > Kind regards,
> > > > Marc-Aur=C3=A8le
> > > >
> > >
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
> >
> >
> > --
> > Rafael Weing=C3=A4rtner
> >
>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

--001a114710a46e3a4005609a3ea7--