cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in finalizeStart
Date Mon, 21 Mar 2016 22:36:33 GMT
To me, it looks like PR #1449 is still relevant since it is minimizing the
number of fetches required when the code is run and PR #1356 just made sure
code was run only when it needed to be.

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Sun, Mar 20, 2016 at 4:55 PM, Remi Bergsma <RBergsma@schubergphilis.com>
wrote:

> Hi Martin,
>
> Thanks, will have a look at it!
>
> These scripts are in the systemvm.iso in the CloudStack release, so not in
> the template. If you build a custom package or war to update your
> management servers you can use it already. Otherwise you need the next
> release.
>
> Regards, Remi
>
> Sent from my iPhone
>
> > On 20 Mar 2016, at 21:36, martin kolly <martin.kolly@senselan.ch> wrote:
> >
> > Hi Remi
> >
> > PR #1449 created as requested. Tests in our environment showed that it
> speeds up the router configuration quite a bit.
> >
> > In the meantime https://github.com/apache/cloudstack/pull/1356 seems to
> be merged which resolves CLOUDSTACK-9255. So not sure if PR#1449 is still
> of interest..
> >
> > Do you know when the system vm with these fixes (PR1356) is available on
> http://cloudstack.apt-get.eu/systemvm/ ?
> >
> > regards
> > Martin
> >
> >
> >> On 03/18/2016 09:45 PM, Remi Bergsma wrote:
> >> Hi Martin,
> >>
> >> Thanks for the fix, didn’t catch you attachment first time.
> >>
> >> Would it be possible for you to send a Pull Request? Is this patch
> against master or a release branch? Generally speaking it’s best to make a
> PR against a release branch, 4.7 would be fine I guess in this case. Once
> it’s a PR we can test it.
> >>
> >> Regards,
> >> Remi
> >>
> >>
> >> From: martin kolly <martin.kolly@senselan.ch<mailto:
> martin.kolly@senselan.ch>>
> >> Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> >> Date: Friday 18 March 2016 at 11:58
> >> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> >> Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
> error in finalizeStart
> >>
> >> Hi All
> >>
> >> We are facing the same issue as reported by Milamber (Ticket 9255)
> https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a
> couple of VMs or Port Forwarding's the re-deployment of the router with
> cleanup fails.
> >>
> >> We found that iptables configuration takes a lot of time, this
> eventually leads to a timeout on the management server "Unable to start VM
> DomainRouter due to error in finalizeStart, not retrying"
> >>
> >> Environment:
> >> - Cloudstack 4.8
> >> - KVM (local storage)
> >> - hosts/mgr on Ubuntu 14.04
> >>
> >> We tested with a simple set of four forwarding rules, here the setup:
> >>
> >> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
> >> {
> >>     "185.20.146.56": [
> >>         {
> >>             "internal_ip": "10.100.1.95",
> >>             "internal_ports": "22:22",
> >>             "protocol": "tcp",
> >>             "public_ip": "185.20.146.56",
> >>             "public_ports": "22:22",
> >>             "type": "forward"
> >>         }
> >>     ],
> >>     "185.20.146.79": [
> >>         {
> >>             "internal_ip": "10.100.1.42",
> >>             "internal_ports": "22:22",
> >>             "protocol": "tcp",
> >>             "public_ip": "185.20.146.79",
> >>             "public_ports": "22:22",
> >>             "type": "forward"
> >>         },
> >>         {
> >>             "internal_ip": "10.100.1.42",
> >>             "internal_ports": "8443:8443",
> >>             "protocol": "tcp",
> >>             "public_ip": "185.20.146.79",
> >>             "public_ports": "8443:8443",
> >>             "type": "forward"
> >>         },
> >>         {
> >>             "internal_ip": "10.100.1.42",
> >>             "internal_ports": "53:53",
> >>             "protocol": "udp",
> >>             "public_ip": "185.20.146.79",
> >>             "public_ports": "53:53",
> >>             "type": "forward"
> >>         }
> >>     ],
> >>     "id": "forwardingrules"
> >>
> >> The definition for every port forwarding seems to take at ~1.5 seconds.
> >>
> >> python /opt/cloud/bin/configure.py.timed
> /etc/cloudstack/forwardingrules.json
> >>
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000965118408203
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.395485162735
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> >> time : 0.395533084869
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 1.16180706024
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.16329216957
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.16407108307
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.53959512711
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000781059265137
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.378201007843
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> >> time : 0.37822508812
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> >> time : 1.14627504349
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j MARK --set-xmark 0x2/0xffffffff
> >> time : 1.1477329731
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14850592613
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52321791649
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000754117965698
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.383729934692
> >> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> >> time : 0.383754968643
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> >> time : 1.14376091957
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.14526605606
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14599299431
> >> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52742600441
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000700950622559
> >> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.382349014282
> >> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> >> time : 0.382384061813
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 1.1425909996
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.14400196075
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14468812943
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52619600296
> >> ----------------------------------------------
> >>
> >> Having a closer look at configure.py how the iptables rules are
> defined. We think that it is not efficient to lookup these values for every
> policy:
> >>
> >> def forward_vr(self, rule):
> >>
> >> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >>               (
> >>                 rule['public_ip'],
> >>                 self.getDeviceByIp(rule['public_ip']),
> >>                 rule['protocol'],
> >>                 rule['protocol'],
> >>                 self.portsToString(rule['public_ports'], ':'),
> >>                 rule['internal_ip'],
> >>                 self.portsToString(rule['internal_ports'], '-')
> >>               )
> >> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >>               (
> >>                 rule['public_ip'],
> >>              self.getDeviceByIp(rule['internal_ip']),
> >>                 rule['protocol'],
> >>                 rule['protocol'],
> >>                 self.portsToString(rule['public_ports'], ':'),
> >>                 rule['internal_ip'],
> >>              self.portsToString(rule['internal_ports'], '-')
> >> .....
> >>
> >>
> >> Defining these values once at the beginning would be much more
> efficient, no ?
> >>
> >> def forward_vr(self, rule):
> >>
> >>        pub_interface = self.getDeviceByIp(rule['public_ip'])
> >>        int_interface = self.getDeviceByIp(rule['internal_ip'])
> >>        pub_ports = self.portsToString(rule['public_ports'], ':')
> >>        int_ports = self.portsToString(rule['internal_ports'], '-')
> >>        int_network = self.getNetworkByIp(rule['internal_ip'])
> >>
> >>  fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >>               (
> >>                 rule['public_ip'],
> >>                 pub_interface,
> >>                 rule['protocol'],
> >>                 rule['protocol'],
> >>                 pub_ports,
> >>                 rule['internal_ip'],
> >>                 int_ports
> >>               )
> >>
> >>  fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >>               (
> >>                 rule['public_ip'],
> >>                 int_interface,
> >>                 rule['protocol'],
> >>                 rule['protocol'],
> >>                 pub_ports,
> >>                 rule['internal_ip'],
> >>                 int_ports
> >>               )
> >> .....
> >>
> >> If we run the configure.py with these modifications we have the
> following:
> >>
> >> root@r-96-VM:~#  python /opt/cloud/bin/configure_modified.py
> /etc/cloudstack/forwardingrules.json
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000349044799805
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000686883926392
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> >> time : 0.000943899154663
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 0.00131487846375
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00161194801331
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00186896324158
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00216102600098
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000232934951782
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000478029251099
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> >> time : 0.00071907043457
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> >> time : 0.000991106033325
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00136613845825
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00174498558044
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00219202041626
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000226974487305
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000502824783325
> >> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> >> time : 0.000762939453125
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> >> time : 0.00103092193604
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00134587287903
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00158596038818
> >> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00182485580444
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000264167785645
> >> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000508069992065
> >> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> >> time : 0.000750064849854
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 0.00102114677429
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00138115882874
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00165915489197
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00196814537048
> >> ----------------------------------------------
> >>
> >> Location of configure.py:
> >>
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
> >>
> >> The modified scripts are attached. Thanks for your feedback.
> >>
> >> regards
> >> Martin
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message