Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of Chiradeep.Vittal@citrix.com
 designates 66.165.176.63 as permitted sender)
From: Chiradeep Vittal <Chiradeep.Vittal@citrix.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Date: Tue, 30 Apr 2013 16:52:40 -0700
Subject: Re: Virtual Router: DHCP and 2-second DNS outages
Thread-Topic: Virtual Router: DHCP and 2-second DNS outages
Thread-Index: Ac5F/dE8jxKIS9cQS2qOrKNlJR5y/w==
Message-ID: <CDA5A655.13C1F%chiradeep.vittal@citrix.com>
In-Reply-To: <040f01ce45f1$b4550a30$1cff1e90$@gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.2.130206
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On 4/30/13 3:26 PM, "Dennis Lawler" <dlawler@gmail.com> wrote:

>Every time a new VM is started up, there is a 2 second outage in DNS
>services that can cause problems in guest VMs that use the router VM for
>DNS.
>
>=20
>
>For Cloudstack configurations using both DHCP and DNS services on the
>router
>VM (both implemented with dnsmasq), there is currently a 2 second DNS
>service outage every time a new VM is instantiated
>
>=20
>
>The source of this outage is in edithosts.sh, which uses "service dnsmasq
>restart" to pick up the freshly added DNS and DHCP entries.
>
>Restarting the dnsmasq service triggers a sleep for 2 seconds after
>killing
>dnsmasq before starting it back up again.
>
>=20
>
>An obvious solution would be to replace "service dnsmasq restart" with
>"kill
>-s 1 $pid" (SIGHUP) so that dnsmasq reads the new DHCP entries without
>restarting, as in dnsmasq_edithosts.sh (external dhcp).
>=20
>
>Unfortunately, this solution is flawed because dnsmasq SIGHUP handling
>does
>not expire in-memory DHCP leases in dnsmasq and all leases are infinite by
>default.

Aha! That's why SIGHUP didn't work consistently. This has been bugging me
for a long time.

>Thus, this will only work if the guest VM performs a DHCP release on
>shutdown, which cannot always be guaranteed.
>
>=20
>
>A few possible solutions off the top of my head:
>
>1.       Separate DNS and DHCP services.  While DHCP services still
>experience an outage during VM,  DNS will not necessarily be impacted if
>implemented correctly.
>
>2.       Use SIGHUP with dnsmasq and implement a removeDhcpEntry interface
>for network appliances to force a DHCP release whenever a NIC / IP is
>deallocated.  This can use dhcp_release to simulate a DHCP release on the
>router VM.
>Catch: dhcp_release is not available for Debian 6.0.  The System VM needs
>to
>be updated to at least Debian 7.0, or the dnsmasq-tools .deb from 7.0
>would
>need to be included in the System VM image.

There is going to be a new system vm based on 7.0 for the upcoming
release. This should work with earlier releases as well.
https://cwiki.apache.org/confluence/x/UlHVAQ

>
>3.       Change DHCP to have a shorter lease, track de-allocation of IPs
>separately from VM destruction.
>Catch: This may cause occasional IP pool exhaustion depending on
>allocation
>of the guest IP range and the rate of VM destruction / instantiation in
>the
>network.
>
>=20
>
>Thoughts?
>