cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Re: Guest VMs not able to acquire DHCP IP from Virtual Router unless Guest and VR are on the same KVM host
Date Tue, 19 Nov 2013 14:12:48 GMT
Thanks Travis. I agree and will see if Corporate IT will comply with that
request. Will post results/response soon.

Best Regards,

Adam Scarcella


On Tue, Nov 19, 2013 at 8:36 AM, Travis Graham <tgraham@tgraham.us> wrote:

> Since DHCP leases aren't something that immediately go away you could
> temporarily stop DHCP services on your default router and see if that
> allows CS instances to pull IPs correctly then enable your default DHCP
> server again once you know more.
>
> That would eliminate your default router from the possible causes list if
> nothing changes or confirm it if things start working correctly in your CS
> environment.
>
> Travis
>
> On Nov 19, 2013, at 8:19 AM, Adam <adam.scarcella@gmail.com> wrote:
>
> > Good Morning David,
> >
> > I'm not sure if you saw my reply from yesterday, but I'm now 99% sure the
> > problem is on our default router/dhcp server on 10.97.38.1. I think we
> did
> > everything we're supposed to to mitigate the normal issues you would
> expect
> > from having two competing DHCP servers on the same network. We went by
> your
> > instructions
> >
> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
> >
> > Perhaps we implemented the filtering incorrectly? My bosses are getting
> > anxious with me and I don't have an answer for them yet.
> >
> > Any further help you could provide would be greatly appreciated. Thanks.
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Mon, Nov 18, 2013 at 12:40 PM, Adam <adam.scarcella@gmail.com> wrote:
> >
> >> Hi David,
> >>
> >> Thanks for the reply.
> >>
> >> I just did some more troubleshooting and I can definitely see using
> >> dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest VM
> >> is on the same KVM Host. If I migrate the guest to any other KVM Host
> that
> >> request never even makes it to the VR. I monitored the physical KVM host
> >> running the VR, but it never registered the DHCPREQUEST regardless of
> where
> >> the guest VM was running.
> >>
> >> I even turned off the firewall on both KVM Hosts and still no
> DHCPREQUEST.
> >>
> >> There actually is another physical DHCP server on the same network.
> *10.97.38.1
> >> Router & DHCP*
> >>
> >> However we (corporate IT Admin) have blocked off a range of IPs
> >> {10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered out
> >> any DHCPREQUESTS made by a '06' MAC address per these instructions
> (written
> >> by you I believe):
> >>
> >>
> >>
> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
> >>
> >> So that tells me that we probably did something wrong on the router and
> >> it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping
> >> those requests from ever reaching the built in VR, but I have no
> visibility
> >> into that unfortunately so I cannot continue to test. I need to be able
> to
> >> give very precise instructions to my IT Admin on what to check.
> >>
> >> Thoughts?
> >>
> >> *dhcpdump from VR when Guest VM is on same KVM Host:*
> >>
> >> root@r-21-VM:~# dhcpdump -i eth0
> >>  TIME: 2013-11-18 08:24:46.997
> >>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> >>    OP: 1 (BOOTPREQUEST)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 060d4c79
> >>  SECS: 0
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 0.0.0.0
> >> SIADDR: 0.0.0.0
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> >> OPTION:  50 (  4) Request IP address        10.97.38.116
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> >>                                             28 (Broadcast address)
> >>                                              2 (Time offset)
> >>                                            121 (Classless Static Route)
> >>                                             15 (Domainname)
> >>                                              6 (DNS server)
> >>                                             12 (Host name)
> >>                                             40 (NIS domain)
> >>                                             41 (NIS servers)
> >>                                             42 (NTP servers)
> >>                                             26 (Interface MTU)
> >>                                            119 (Domain Search)
> >>                                              3 (Routers)
> >>                                            121 (Classless Static Route)
> >>                                            249 (MSFT - Classless route)
> >>                                             42 (NTP servers)
> >>
> >>
> ---------------------------------------------------------------------------
> >>
> >>  TIME: 2013-11-18 08:24:53.998
> >>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> >>    OP: 1 (BOOTPREQUEST)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 060d4c79
> >>  SECS: 7
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 0.0.0.0
> >> SIADDR: 0.0.0.0
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> >> OPTION:  50 (  4) Request IP address        10.97.38.116
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> >>                                             28 (Broadcast address)
> >>                                              2 (Time offset)
> >>                                            121 (Classless Static Route)
> >>                                             15 (Domainname)
> >>                                              6 (DNS server)
> >>                                             12 (Host name)
> >>                                             40 (NIS domain)
> >>                                             41 (NIS servers)
> >>                                             42 (NTP servers)
> >>                                             26 (Interface MTU)
> >>                                            119 (Domain Search)
> >>                                              3 (Routers)
> >>                                            121 (Classless Static Route)
> >>                                            249 (MSFT - Classless route)
> >>                                             42 (NTP servers)
> >>
> >>
> ---------------------------------------------------------------------------
> >>
> >>  TIME: 2013-11-18 08:25:02.003
> >>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> >>    OP: 1 (BOOTPREQUEST)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 91cdcd74
> >>  SECS: 0
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 0.0.0.0
> >> SIADDR: 0.0.0.0
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         1 (DHCPDISCOVER)
> >> OPTION:  50 (  4) Request IP address        10.97.38.116
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> >>                                             28 (Broadcast address)
> >>                                              2 (Time offset)
> >>                                            121 (Classless Static Route)
> >>                                             15 (Domainname)
> >>                                              6 (DNS server)
> >>                                             12 (Host name)
> >>                                             40 (NIS domain)
> >>                                             41 (NIS servers)
> >>                                             42 (NTP servers)
> >>                                             26 (Interface MTU)
> >>                                            119 (Domain Search)
> >>                                              3 (Routers)
> >>                                            121 (Classless Static Route)
> >>                                            249 (MSFT - Classless route)
> >>                                             42 (NTP servers)
> >>
> >>
> ---------------------------------------------------------------------------
> >>
> >>  TIME: 2013-11-18 08:25:02.014
> >>    IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
> >>    OP: 2 (BOOTPREPLY)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 91cdcd74
> >>  SECS: 0
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 10.97.38.116
> >> SIADDR: 10.97.38.113
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         2 (DHCPOFFER)
> >> OPTION:  54 (  4) Server identifier         10.97.38.113
> >> OPTION:  51 (  4) IP address leasetime      -1 ()
> >> OPTION:   1 (  4) Subnet mask               255.255.255.0
> >> OPTION:  28 (  4) Broadcast address         10.97.38.255
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
> >> OPTION:   3 (  4) Routers                   10.97.38.1
> >> OPTION:  15 ( 17) Domainname                cs1cloud.internal
> >>
> ---------------------------------------------------------------------------
> >>
> >>  TIME: 2013-11-18 08:25:02.015
> >>    IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> >>    OP: 1 (BOOTPREQUEST)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 91cdcd74
> >>  SECS: 0
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 0.0.0.0
> >> SIADDR: 0.0.0.0
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> >> OPTION:  54 (  4) Server identifier         10.97.38.113
> >> OPTION:  50 (  4) Request IP address        10.97.38.116
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> >>                                             28 (Broadcast address)
> >>                                              2 (Time offset)
> >>                                            121 (Classless Static Route)
> >>                                             15 (Domainname)
> >>                                              6 (DNS server)
> >>                                             12 (Host name)
> >>                                             40 (NIS domain)
> >>                                             41 (NIS servers)
> >>                                             42 (NTP servers)
> >>                                             26 (Interface MTU)
> >>                                            119 (Domain Search)
> >>                                              3 (Routers)
> >>                                            121 (Classless Static Route)
> >>                                            249 (MSFT - Classless route)
> >>                                             42 (NTP servers)
> >>
> >>
> ---------------------------------------------------------------------------
> >>
> >>  TIME: 2013-11-18 08:25:02.016
> >>    IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
> >>    OP: 2 (BOOTPREPLY)
> >> HTYPE: 1 (Ethernet)
> >>  HLEN: 6
> >>  HOPS: 0
> >>   XID: 91cdcd74
> >>  SECS: 0
> >> FLAGS: 0
> >> CIADDR: 0.0.0.0
> >> YIADDR: 10.97.38.116
> >> SIADDR: 10.97.38.113
> >> GIADDR: 0.0.0.0
> >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> >> SNAME: .
> >> FNAME: .
> >> OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
> >> OPTION:  54 (  4) Server identifier         10.97.38.113
> >> OPTION:  51 (  4) IP address leasetime      -1 ()
> >> OPTION:   1 (  4) Subnet mask               255.255.255.0
> >> OPTION:  28 (  4) Broadcast address         10.97.38.255
> >> OPTION:  12 ( 16) Host name                 centos6-template
> >> OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
> >> OPTION:   3 (  4) Routers                   10.97.38.1
> >> OPTION:  15 ( 17) Domainname                cs1cloud.internal
> >>
> ---------------------------------------------------------------------------
> >>
> >> root@r-21-VM:~#
> >>
> >>
> >> Best Regards,
> >>
> >>
> >>
> >> Adam Scarcella
> >>
> >>
> >> On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <david@gnsa.us> wrote:
> >>
> >>> So having seen this specific problem scores of times, it's almost
> >>> always network config related. Several potential problems to look out
> >>> for:
> >>>
> >>> 1. Another DHCP server on the same network. (There are ways of
> >>> mitigating this, but for the time being, lets just say that
> >>> CloudStack's VR should be the only DHCP server on the network.
> >>>
> >>> 2. Shell into the physical machine that the VR is running on. Using
> >>> tcpdump (with filters of course) Do you see the broadcast asking for a
> >>> DHCP address coming from the VM on the other physical machine? Do you
> >>> see the VR answering the request? If so, move to step 3. If not (for
> >>> either of those questions) - you aren't communicating between the two
> >>> physical hosts - your switch config is suspect.
> >>>
> >>> 3. Shell into the physical machine that the VM is running on - You've
> >>> seen the response with an IP address assigned go out - so see if it is
> >>> seen by the physical host NIC? If not, you've again. If you see the
> >>> response (and are sure it's from the VR and not another DHCP server)
> >>> come back and tell us - something funky is going on.
> >>>
> >>> --David
> >>>
> >>> On Sun, Nov 17, 2013 at 10:11 AM, Adam <adam.scarcella@gmail.com>
> wrote:
> >>>> Yes, I've disabled the firewall on both KVM hosts in question and
> still
> >>> no
> >>>> dice. I can't even ping the VR from my guest VM, but when I set the
> eth0
> >>>> device on the guest to static everything works fine, which makes no
> >>> sense
> >>>> at all to me. Simply setting the NIC to static allows me see the VR.
> >>>> Switching back to DHCP kills it again. I do not understand what is
> >>> required
> >>>> for the guests to acquire a DHCP lease from the VR. I know that the
VR
> >>> is
> >>>> running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more
> info,
> >>>> but only see a DHCP request when the guest is on the same KVM host as
> >>> the
> >>>> VR.
> >>>>
> >>>> Does anyone know exactly how to troubleshoot this scenario? I'm not
> even
> >>>> sure what to look for.
> >>>>
> >>>> -Adam
> >>>>
> >>>> Best Regards,
> >>>>
> >>>>
> >>>>
> >>>> Adam Scarcella
> >>>>
> >>>>
> >>>> On Sat, Nov 16, 2013 at 5:43 PM, Carlos ReƔtegui <creategui@gmail.com
> >>>> wrote:
> >>>>
> >>>>> Did you also check the host firewall?  Try disabling.
> >>>>>
> >>>>>> On Nov 16, 2013, at 1:51 PM, Adam <adam.scarcella@gmail.com>
wrote:
> >>>>>>
> >>>>>> The hosts have static IPs (10.97.38.[10-14]) and can all see
and
> >>> talk to
> >>>>>> each other via IP and hostname. I'm only using a basic zone
so no
> >>> VLAN
> >>>>>> tagging or anything funky like that.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Adam Scarcella
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky <
> >>> andrei@arhont.com
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Adam, it sounds like a networking issue, check that all
hosts can
> >>> talk
> >>>>> to
> >>>>>>> each other on the same vlan that is used for guest network.
I had
> >>> the
> >>>>> same
> >>>>>>> issue with advanced networking when my vlans were not properly
> >>> setup on
> >>>>> the
> >>>>>>> switches.
> >>>>>>>
> >>>>>>> Andrei
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>
> >>>>>>> From: "Adam" <adam.scarcella@gmail.com>
> >>>>>>> To: users@cloudstack.apache.org
> >>>>>>> Sent: Saturday, 16 November, 2013 7:08:50 PM
> >>>>>>> Subject: Guest VMs not able to acquire DHCP IP from Virtual
Router
> >>>>> unless
> >>>>>>> Guest and VR are on the same KVM host
> >>>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> I have a new and very strange issue that for the life of
me I
> cannot
> >>>>> seem
> >>>>>>> to track down and fix.
> >>>>>>>
> >>>>>>> I have CS 4.2 running on 5 hosts in a simple basic zone.
All is
> >>> working
> >>>>>>> fine, except that my Guest VMs cannot seem to get a DHCP
lease from
> >>> the
> >>>>>>> Virtual Router unless I migrate the Guest VM to the same
physical
> >>> KVM
> >>>>>>> Host running the VR. That would seem to indicate a firewall
issue,
> >>> but
> >>>>> I've
> >>>>>>> tested by turning off both the firewalls (VR KVM Host iptables
&
> >>> Guest
> >>>>> VM
> >>>>>>> KVM Host iptables). It didn't help. The only way to fix
it is to
> >>> migrate
> >>>>>>> the Guest VM to the same KVM Host that's running the Virtual
> Router.
> >>>>>>>
> >>>>>>> NOTE: The Console Proxy has worked flawlessly this whole
time.
> >>>>>>>
> >>>>>>> So, if a Guest VM starts on a different physical KVM Host,
it will
> >>> not
> >>>>> get
> >>>>>>> an internal IP of 169.254.x.x. All along the Console Proxy
works
> >>> fine.
> >>>>> Then
> >>>>>>> if I migrate the Guest VM to the same KVM host that's running
the
> >>> VR,
> >>>>> DHCP
> >>>>>>> automatically starts working and the Guest VM receives a
proper IP
> >>>>> address
> >>>>>>> of 10.97.38.x. Then I can migrate the Guest VM back to any
other
> >>>>> physical
> >>>>>>> KVM Host and believe it or not, it continues to work flawlessly,
> >>> until I
> >>>>>>> either reboot the VM or restart the network services. Then
it
> >>> cannot see
> >>>>>>> the VR again and instead receives an internal IP of 169.254.x.x.
If
> >>> I
> >>>>> set a
> >>>>>>> static IP address & DNS everything works fine, no matter
where the
> >>>>> Guest VM
> >>>>>>> is running.
> >>>>>>>
> >>>>>>> Setting static IPs is not an option
> >>>>>>> Running all the Guest VMs on the same physical KVM host
is not an
> >>> option
> >>>>>>>
> >>>>>>> I desperately need to track down the root cause of this
issue so I
> >>> can
> >>>>>>> release this cloud to my entire department by Monday morning.
> >>> Someone
> >>>>>>> please help!
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Adam Scarcella
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message