Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Thu, 4 May 2017 10:48:42 +0200 (CEST)
From: Wido den Hollander <wido@widodh.nl>
To: Daan Hoogland <daan.hoogland@gmail.com>, dev@cloudstack.apache.org
Message-ID: <1327248679.3490.1493887722616@ox.pcextreme.nl>
In-Reply-To: <CAGQtxvayPUX73SFLvKY75nxNTFqBMzRfL40q+vUHjKDZRo7vLg@mail.gmail.com>
References: <2032837791.3395.1493747829105@ox.pcextreme.nl>
 <DB6PR0701MB301632E63532C31B0216215EB2160@DB6PR0701MB3016.eurprd07.prod.outlook.com>
 <CADOTMGF_odwbqR1u4P8-76VbCghR8nuPDTtdB7L9bTDPWciDDw@mail.gmail.com>
 <8E18DC0D-BC0A-4EB2-878A-607AA0EE664F@schubergphilis.com>
 <DB5PR07MB1685D88F00421221D1B00CCDE9160@DB5PR07MB1685.eurprd07.prod.outlook.com>
 <F05CA40C-D86C-4186-BFCE-34EC0B7EC729@schubergphilis.com>
 <CAGQtxvayPUX73SFLvKY75nxNTFqBMzRfL40q+vUHjKDZRo7vLg@mail.gmail.com>
Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Importance: Medium
archived-at: Thu, 04 May 2017 08:49:00 -0000

Thanks Daan, Remi.

I found a additional bug where it seems that 'network.dns.basiczone.updates=
' isn't read when sending DHCP settings in Basic Networking.

This means that the VR gets all DHCP setting for the whole zone instead of =
just for that POD.

In this case some VRs we have get ~2k of DHCP offerings send to them which =
causes a large slowdown.

Wido

> Op 3 mei 2017 om 14:49 schreef Daan Hoogland <daan.hoogland@gmail.com>:
>=20
>=20
> Happy to pick this up, Remi. I'm travelling now but will look at both on
> Friday.
>=20
> Biligual auto correct use.  Read at your own risico
>=20
> On 3 May 2017 2:25 pm, "Remi Bergsma" <RBergsma@schubergphilis.com> wrote=
:
>=20
> > Always happy to share, but I won=E2=80=99t have time to work on porting=
 this to
> > CloudStack any time soon.
> >
> > Regards, Remi
> >
> >
> > On 03/05/2017, 13:44, "Rohit Yadav" <rohit.yadav@shapeblue.com> wrote:
> >
> >     Hi Remi, thanks for sharing. We would love to have those changes (f=
or
> > 4.9+), looking forward to your pull requests.
> >
> >
> >     Regards.
> >
> >     ________________________________
> >     From: Remi Bergsma <RBergsma@schubergphilis.com>
> >     Sent: 03 May 2017 16:58:18
> >     To: dev@cloudstack.apache.org
> >     Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0
> >
> >     Hi,
> >
> >     The patches I talked about:
> >
> >     1) Iptables speed improvement
> >     https://github.com/apache/cloudstack/pull/1482
> >     Was reverted due to a licensing issue.
> >
> >     2) Passwd speed improvement
> >     https://github.com/MissionCriticalCloudOldRepos/cosmic-core/pull/13=
8
> >
> >     By now, these are rather old patches so they need some work before
> > they apply to CloudStack again.
> >
> >     Regards, Remi
> >
> >
> >
> >     On 03/05/2017, 12:49, "Jeff Hair" <jeff@greenqloud.com> wrote:
> >
> >         Hi Remi,
> >
> >         Do you have a link to the PR that was reverted? And also possib=
ly
> > the code
> >         that makes the password updating more efficient?
> >
> >         Jeff
> >
> >         On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma <
> > RBergsma@schubergphilis.com>
> >         wrote:
> >
> >         > Hi Wido,
> >         >
> >         > When we had similar issues last year, we found that for examp=
le
> > comparing
> >         > the iptables rules one-by-one is 1000x slower than simply
> > loading them all
> >         > at once. Boris rewrote this part in our Cosmic fork, may be
> > worth looking
> >         > into this again. The PR to CloudStack was merged, but reverte=
d
> > later, can't
> >         > remember why. We run it in production ever since. Also feedin=
g
> > passwords to
> >         > the passwd server is very inefficient (it operates like a
> > snowball and gets
> >         > slower once you have more VMs). That we also fixed in Cosmic,
> > not sure if
> >         > that patch made it upstream. Wrote it about a year ago alread=
y.
> >         >
> >         > We tested applying 10K iptables rules in just a couple of
> > seconds. 1000
> >         > VMs takes a few minutes to deploy.
> >         >
> >         > Generally speaking I'd suggest looking at the logs to find wh=
at
> > takes long
> >         > or is executed a lot of times. Iptables and passwd are two to
> > look at.
> >         >
> >         > If you want I can lookup the patches. Not handy on my phone n=
ow
> > ;-)
> >         >
> >         > Regards, Remi
> >         > ________________________________
> >         > From: Wido den Hollander <wido@widodh.nl>
> >         > Sent: Tuesday, May 2, 2017 7:57:08 PM
> >         > To: dev@cloudstack.apache.org
> >         > Subject: Very slow Virtual Router provisioning with 4.9.2.0
> >         >
> >         > Hi,
> >         >
> >         > Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. Al=
l
> > went well,
> >         > but the VR provisioning is terribly slow which causes all kin=
ds
> > of problems.
> >         >
> >         > The vr_cfg.sh and update_config.py scripts start to run. Rest=
art
> > dnsmasq,
> >         > add metadata, etc.
> >         >
> >         > But for just 1800 hosts this can take up to 2 hours and that
> > causes
> >         > timeouts in the management server and other problems.
> >         >
> >         > 2 hours is just very, very slow. So I am starting to wonder i=
f
> > something
> >         > is wrong here.
> >         >
> >         > Did anybody else see this?
> >         >
> >         > Running Basic Networking with CloudStack 4.9.2.0
> >         >
> >         > Wido
> >         >
> >
> >
> >
> >
> >     rohit.yadav@shapeblue.com
> >     www.shapeblue.com
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     @shapeblue
> >
> >
> >
> >
> >
> >