cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remi Bergsma <RBerg...@schubergphilis.com>
Subject Re: Very slow Virtual Router provisioning with 4.9.2.0
Date Wed, 03 May 2017 10:36:35 GMT
Hi Wido,

When we had similar issues last year, we found that for example comparing the iptables rules
one-by-one is 1000x slower than simply loading them all at once. Boris rewrote this part in
our Cosmic fork, may be worth looking into this again. The PR to CloudStack was merged, but
reverted later, can't remember why. We run it in production ever since. Also feeding passwords
to the passwd server is very inefficient (it operates like a snowball and gets slower once
you have more VMs). That we also fixed in Cosmic, not sure if that patch made it upstream.
Wrote it about a year ago already.

We tested applying 10K iptables rules in just a couple of seconds. 1000 VMs takes a few minutes
to deploy.

Generally speaking I'd suggest looking at the logs to find what takes long or is executed
a lot of times. Iptables and passwd are two to look at.

If you want I can lookup the patches. Not handy on my phone now ;-)

Regards, Remi
________________________________
From: Wido den Hollander <wido@widodh.nl>
Sent: Tuesday, May 2, 2017 7:57:08 PM
To: dev@cloudstack.apache.org
Subject: Very slow Virtual Router provisioning with 4.9.2.0

Hi,

Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well, but the VR provisioning
is terribly slow which causes all kinds of problems.

The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq, add metadata, etc.

But for just 1800 hosts this can take up to 2 hours and that causes timeouts in the management
server and other problems.

2 hours is just very, very slow. So I am starting to wonder if something is wrong here.

Did anybody else see this?

Running Basic Networking with CloudStack 4.9.2.0

Wido

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message