cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Graham <tgra...@tgraham.us>
Subject Re: Guest VMs not able to acquire DHCP IP from Virtual Router unless Guest and VR are on the same KVM host
Date Tue, 19 Nov 2013 13:36:30 GMT
Since DHCP leases aren't something that immediately go away you could temporarily stop DHCP
services on your default router and see if that allows CS instances to pull IPs correctly
then enable your default DHCP server again once you know more.

That would eliminate your default router from the possible causes list if nothing changes
or confirm it if things start working correctly in your CS environment.

Travis

On Nov 19, 2013, at 8:19 AM, Adam <adam.scarcella@gmail.com> wrote:

> Good Morning David,
> 
> I'm not sure if you saw my reply from yesterday, but I'm now 99% sure the
> problem is on our default router/dhcp server on 10.97.38.1. I think we did
> everything we're supposed to to mitigate the normal issues you would expect
> from having two competing DHCP servers on the same network. We went by your
> instructions
> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
> 
> Perhaps we implemented the filtering incorrectly? My bosses are getting
> anxious with me and I don't have an answer for them yet.
> 
> Any further help you could provide would be greatly appreciated. Thanks.
> 
> Best Regards,
> 
> 
> 
> Adam Scarcella
> 
> 
> On Mon, Nov 18, 2013 at 12:40 PM, Adam <adam.scarcella@gmail.com> wrote:
> 
>> Hi David,
>> 
>> Thanks for the reply.
>> 
>> I just did some more troubleshooting and I can definitely see using
>> dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest VM
>> is on the same KVM Host. If I migrate the guest to any other KVM Host that
>> request never even makes it to the VR. I monitored the physical KVM host
>> running the VR, but it never registered the DHCPREQUEST regardless of where
>> the guest VM was running.
>> 
>> I even turned off the firewall on both KVM Hosts and still no DHCPREQUEST.
>> 
>> There actually is another physical DHCP server on the same network. *10.97.38.1
>> Router & DHCP*
>> 
>> However we (corporate IT Admin) have blocked off a range of IPs
>> {10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered out
>> any DHCPREQUESTS made by a '06' MAC address per these instructions (written
>> by you I believe):
>> 
>> 
>> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
>> 
>> So that tells me that we probably did something wrong on the router and
>> it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping
>> those requests from ever reaching the built in VR, but I have no visibility
>> into that unfortunately so I cannot continue to test. I need to be able to
>> give very precise instructions to my IT Admin on what to check.
>> 
>> Thoughts?
>> 
>> *dhcpdump from VR when Guest VM is on same KVM Host:*
>> 
>> root@r-21-VM:~# dhcpdump -i eth0
>>  TIME: 2013-11-18 08:24:46.997
>>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
>>    OP: 1 (BOOTPREQUEST)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 060d4c79
>>  SECS: 0
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 0.0.0.0
>> SIADDR: 0.0.0.0
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
>> OPTION:  50 (  4) Request IP address        10.97.38.116
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
>>                                             28 (Broadcast address)
>>                                              2 (Time offset)
>>                                            121 (Classless Static Route)
>>                                             15 (Domainname)
>>                                              6 (DNS server)
>>                                             12 (Host name)
>>                                             40 (NIS domain)
>>                                             41 (NIS servers)
>>                                             42 (NTP servers)
>>                                             26 (Interface MTU)
>>                                            119 (Domain Search)
>>                                              3 (Routers)
>>                                            121 (Classless Static Route)
>>                                            249 (MSFT - Classless route)
>>                                             42 (NTP servers)
>> 
>> ---------------------------------------------------------------------------
>> 
>>  TIME: 2013-11-18 08:24:53.998
>>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
>>    OP: 1 (BOOTPREQUEST)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 060d4c79
>>  SECS: 7
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 0.0.0.0
>> SIADDR: 0.0.0.0
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
>> OPTION:  50 (  4) Request IP address        10.97.38.116
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
>>                                             28 (Broadcast address)
>>                                              2 (Time offset)
>>                                            121 (Classless Static Route)
>>                                             15 (Domainname)
>>                                              6 (DNS server)
>>                                             12 (Host name)
>>                                             40 (NIS domain)
>>                                             41 (NIS servers)
>>                                             42 (NTP servers)
>>                                             26 (Interface MTU)
>>                                            119 (Domain Search)
>>                                              3 (Routers)
>>                                            121 (Classless Static Route)
>>                                            249 (MSFT - Classless route)
>>                                             42 (NTP servers)
>> 
>> ---------------------------------------------------------------------------
>> 
>>  TIME: 2013-11-18 08:25:02.003
>>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
>>    OP: 1 (BOOTPREQUEST)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 91cdcd74
>>  SECS: 0
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 0.0.0.0
>> SIADDR: 0.0.0.0
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         1 (DHCPDISCOVER)
>> OPTION:  50 (  4) Request IP address        10.97.38.116
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
>>                                             28 (Broadcast address)
>>                                              2 (Time offset)
>>                                            121 (Classless Static Route)
>>                                             15 (Domainname)
>>                                              6 (DNS server)
>>                                             12 (Host name)
>>                                             40 (NIS domain)
>>                                             41 (NIS servers)
>>                                             42 (NTP servers)
>>                                             26 (Interface MTU)
>>                                            119 (Domain Search)
>>                                              3 (Routers)
>>                                            121 (Classless Static Route)
>>                                            249 (MSFT - Classless route)
>>                                             42 (NTP servers)
>> 
>> ---------------------------------------------------------------------------
>> 
>>  TIME: 2013-11-18 08:25:02.014
>>    IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
>>    OP: 2 (BOOTPREPLY)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 91cdcd74
>>  SECS: 0
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 10.97.38.116
>> SIADDR: 10.97.38.113
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         2 (DHCPOFFER)
>> OPTION:  54 (  4) Server identifier         10.97.38.113
>> OPTION:  51 (  4) IP address leasetime      -1 ()
>> OPTION:   1 (  4) Subnet mask               255.255.255.0
>> OPTION:  28 (  4) Broadcast address         10.97.38.255
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
>> OPTION:   3 (  4) Routers                   10.97.38.1
>> OPTION:  15 ( 17) Domainname                cs1cloud.internal
>> ---------------------------------------------------------------------------
>> 
>>  TIME: 2013-11-18 08:25:02.015
>>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
>>    OP: 1 (BOOTPREQUEST)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 91cdcd74
>>  SECS: 0
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 0.0.0.0
>> SIADDR: 0.0.0.0
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
>> OPTION:  54 (  4) Server identifier         10.97.38.113
>> OPTION:  50 (  4) Request IP address        10.97.38.116
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
>>                                             28 (Broadcast address)
>>                                              2 (Time offset)
>>                                            121 (Classless Static Route)
>>                                             15 (Domainname)
>>                                              6 (DNS server)
>>                                             12 (Host name)
>>                                             40 (NIS domain)
>>                                             41 (NIS servers)
>>                                             42 (NTP servers)
>>                                             26 (Interface MTU)
>>                                            119 (Domain Search)
>>                                              3 (Routers)
>>                                            121 (Classless Static Route)
>>                                            249 (MSFT - Classless route)
>>                                             42 (NTP servers)
>> 
>> ---------------------------------------------------------------------------
>> 
>>  TIME: 2013-11-18 08:25:02.016
>>    IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
>>    OP: 2 (BOOTPREPLY)
>> HTYPE: 1 (Ethernet)
>>  HLEN: 6
>>  HOPS: 0
>>   XID: 91cdcd74
>>  SECS: 0
>> FLAGS: 0
>> CIADDR: 0.0.0.0
>> YIADDR: 10.97.38.116
>> SIADDR: 10.97.38.113
>> GIADDR: 0.0.0.0
>> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
>> SNAME: .
>> FNAME: .
>> OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
>> OPTION:  54 (  4) Server identifier         10.97.38.113
>> OPTION:  51 (  4) IP address leasetime      -1 ()
>> OPTION:   1 (  4) Subnet mask               255.255.255.0
>> OPTION:  28 (  4) Broadcast address         10.97.38.255
>> OPTION:  12 ( 16) Host name                 centos6-template
>> OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
>> OPTION:   3 (  4) Routers                   10.97.38.1
>> OPTION:  15 ( 17) Domainname                cs1cloud.internal
>> ---------------------------------------------------------------------------
>> 
>> root@r-21-VM:~#
>> 
>> 
>> Best Regards,
>> 
>> 
>> 
>> Adam Scarcella
>> 
>> 
>> On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <david@gnsa.us> wrote:
>> 
>>> So having seen this specific problem scores of times, it's almost
>>> always network config related. Several potential problems to look out
>>> for:
>>> 
>>> 1. Another DHCP server on the same network. (There are ways of
>>> mitigating this, but for the time being, lets just say that
>>> CloudStack's VR should be the only DHCP server on the network.
>>> 
>>> 2. Shell into the physical machine that the VR is running on. Using
>>> tcpdump (with filters of course) Do you see the broadcast asking for a
>>> DHCP address coming from the VM on the other physical machine? Do you
>>> see the VR answering the request? If so, move to step 3. If not (for
>>> either of those questions) - you aren't communicating between the two
>>> physical hosts - your switch config is suspect.
>>> 
>>> 3. Shell into the physical machine that the VM is running on - You've
>>> seen the response with an IP address assigned go out - so see if it is
>>> seen by the physical host NIC? If not, you've again. If you see the
>>> response (and are sure it's from the VR and not another DHCP server)
>>> come back and tell us - something funky is going on.
>>> 
>>> --David
>>> 
>>> On Sun, Nov 17, 2013 at 10:11 AM, Adam <adam.scarcella@gmail.com> wrote:
>>>> Yes, I've disabled the firewall on both KVM hosts in question and still
>>> no
>>>> dice. I can't even ping the VR from my guest VM, but when I set the eth0
>>>> device on the guest to static everything works fine, which makes no
>>> sense
>>>> at all to me. Simply setting the NIC to static allows me see the VR.
>>>> Switching back to DHCP kills it again. I do not understand what is
>>> required
>>>> for the guests to acquire a DHCP lease from the VR. I know that the VR
>>> is
>>>> running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more info,
>>>> but only see a DHCP request when the guest is on the same KVM host as
>>> the
>>>> VR.
>>>> 
>>>> Does anyone know exactly how to troubleshoot this scenario? I'm not even
>>>> sure what to look for.
>>>> 
>>>> -Adam
>>>> 
>>>> Best Regards,
>>>> 
>>>> 
>>>> 
>>>> Adam Scarcella
>>>> 
>>>> 
>>>> On Sat, Nov 16, 2013 at 5:43 PM, Carlos ReƔtegui <creategui@gmail.com
>>>> wrote:
>>>> 
>>>>> Did you also check the host firewall?  Try disabling.
>>>>> 
>>>>>> On Nov 16, 2013, at 1:51 PM, Adam <adam.scarcella@gmail.com>
wrote:
>>>>>> 
>>>>>> The hosts have static IPs (10.97.38.[10-14]) and can all see and
>>> talk to
>>>>>> each other via IP and hostname. I'm only using a basic zone so no
>>> VLAN
>>>>>> tagging or anything funky like that.
>>>>>> 
>>>>>> Best Regards,
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Adam Scarcella
>>>>>> 
>>>>>> 
>>>>>> On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky <
>>> andrei@arhont.com
>>>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Adam, it sounds like a networking issue, check that all hosts
can
>>> talk
>>>>> to
>>>>>>> each other on the same vlan that is used for guest network. I
had
>>> the
>>>>> same
>>>>>>> issue with advanced networking when my vlans were not properly
>>> setup on
>>>>> the
>>>>>>> switches.
>>>>>>> 
>>>>>>> Andrei
>>>>>>> 
>>>>>>> ----- Original Message -----
>>>>>>> 
>>>>>>> From: "Adam" <adam.scarcella@gmail.com>
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Sent: Saturday, 16 November, 2013 7:08:50 PM
>>>>>>> Subject: Guest VMs not able to acquire DHCP IP from Virtual Router
>>>>> unless
>>>>>>> Guest and VR are on the same KVM host
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> I have a new and very strange issue that for the life of me I
cannot
>>>>> seem
>>>>>>> to track down and fix.
>>>>>>> 
>>>>>>> I have CS 4.2 running on 5 hosts in a simple basic zone. All
is
>>> working
>>>>>>> fine, except that my Guest VMs cannot seem to get a DHCP lease
from
>>> the
>>>>>>> Virtual Router unless I migrate the Guest VM to the same physical
>>> KVM
>>>>>>> Host running the VR. That would seem to indicate a firewall issue,
>>> but
>>>>> I've
>>>>>>> tested by turning off both the firewalls (VR KVM Host iptables
&
>>> Guest
>>>>> VM
>>>>>>> KVM Host iptables). It didn't help. The only way to fix it is
to
>>> migrate
>>>>>>> the Guest VM to the same KVM Host that's running the Virtual
Router.
>>>>>>> 
>>>>>>> NOTE: The Console Proxy has worked flawlessly this whole time.
>>>>>>> 
>>>>>>> So, if a Guest VM starts on a different physical KVM Host, it
will
>>> not
>>>>> get
>>>>>>> an internal IP of 169.254.x.x. All along the Console Proxy works
>>> fine.
>>>>> Then
>>>>>>> if I migrate the Guest VM to the same KVM host that's running
the
>>> VR,
>>>>> DHCP
>>>>>>> automatically starts working and the Guest VM receives a proper
IP
>>>>> address
>>>>>>> of 10.97.38.x. Then I can migrate the Guest VM back to any other
>>>>> physical
>>>>>>> KVM Host and believe it or not, it continues to work flawlessly,
>>> until I
>>>>>>> either reboot the VM or restart the network services. Then it
>>> cannot see
>>>>>>> the VR again and instead receives an internal IP of 169.254.x.x.
If
>>> I
>>>>> set a
>>>>>>> static IP address & DNS everything works fine, no matter
where the
>>>>> Guest VM
>>>>>>> is running.
>>>>>>> 
>>>>>>> Setting static IPs is not an option
>>>>>>> Running all the Guest VMs on the same physical KVM host is not
an
>>> option
>>>>>>> 
>>>>>>> I desperately need to track down the root cause of this issue
so I
>>> can
>>>>>>> release this cloud to my entire department by Monday morning.
>>> Someone
>>>>>>> please help!
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Adam Scarcella
>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> 
>> 


Mime
View raw message