cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Re: Guest VMs not able to acquire DHCP IP from Virtual Router unless Guest and VR are on the same KVM host
Date Tue, 19 Nov 2013 22:58:29 GMT
Hi Carlos,

Thanks for the reply. That is exactly what I've requested, but our
CloudStack project is a grass-roots POC and hasn't actually received proper
"wings" yet, which is a catch-22 situation since if I can't prove how
useful and effective CS is than it probably won't take flight, but if they
don't give me my own /24 VLAN for this POC, then I probably won't be able
to fully demonstrate how powerful CS really is.

I'll keep pushing though. I really want this to work so that I can
ultimately build a large multi-regional CS cloud with 4 regions and
multiple zones per region for out Global Support, QC and Engineering teams.
I figure if we can master Puppet and configure this such that everyone who
uses it sees almost instantly that they can't live without it, then it will
absolutely take flight.

For this particular silly DHCP issue, I will have to continue to push for
our own /24 VLAN.

Best Regards,



Adam Scarcella


On Tue, Nov 19, 2013 at 4:29 PM, Carlos Reategui <carlos@reategui.com>wrote:

> Adam,
> Any chance of getting your own /24 VLAN without DHCP?  Then you just need a
> switch that you control with all your machines hanging off of it connected
> to that VLAN.  As long as your IT sets up the routing so that you can get
> to the rest of the network (and out) you should be ok.  It may make things
> easier for you.  That is basically what I have.
>
> Regards,
> Carlos
>
>
> On Tue, Nov 19, 2013 at 5:19 AM, Adam <adam.scarcella@gmail.com> wrote:
>
> > Good Morning David,
> >
> > I'm not sure if you saw my reply from yesterday, but I'm now 99% sure the
> > problem is on our default router/dhcp server on 10.97.38.1. I think we
> did
> > everything we're supposed to to mitigate the normal issues you would
> expect
> > from having two competing DHCP servers on the same network. We went by
> your
> > instructions
> >
> >
> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
> >
> > Perhaps we implemented the filtering incorrectly? My bosses are getting
> > anxious with me and I don't have an answer for them yet.
> >
> > Any further help you could provide would be greatly appreciated. Thanks.
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Mon, Nov 18, 2013 at 12:40 PM, Adam <adam.scarcella@gmail.com> wrote:
> >
> > > Hi David,
> > >
> > > Thanks for the reply.
> > >
> > > I just did some more troubleshooting and I can definitely see using
> > > dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest
> VM
> > > is on the same KVM Host. If I migrate the guest to any other KVM Host
> > that
> > > request never even makes it to the VR. I monitored the physical KVM
> host
> > > running the VR, but it never registered the DHCPREQUEST regardless of
> > where
> > > the guest VM was running.
> > >
> > > I even turned off the firewall on both KVM Hosts and still no
> > DHCPREQUEST.
> > >
> > > There actually is another physical DHCP server on the same network.
> > *10.97.38.1
> > > Router & DHCP*
> > >
> > > However we (corporate IT Admin) have blocked off a range of IPs
> > > {10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered
> out
> > > any DHCPREQUESTS made by a '06' MAC address per these instructions
> > (written
> > > by you I believe):
> > >
> > >
> > >
> >
> http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
> > >
> > > So that tells me that we probably did something wrong on the router and
> > > it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping
> > > those requests from ever reaching the built in VR, but I have no
> > visibility
> > > into that unfortunately so I cannot continue to test. I need to be able
> > to
> > > give very precise instructions to my IT Admin on what to check.
> > >
> > > Thoughts?
> > >
> > > *dhcpdump from VR when Guest VM is on same KVM Host:*
> > >
> > > root@r-21-VM:~# dhcpdump -i eth0
> > >   TIME: 2013-11-18 08:24:46.997
> > >     IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> > >     OP: 1 (BOOTPREQUEST)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 060d4c79
> > >   SECS: 0
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 0.0.0.0
> > > SIADDR: 0.0.0.0
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> > > OPTION:  50 (  4) Request IP address        10.97.38.116
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> > >                                              28 (Broadcast address)
> > >                                               2 (Time offset)
> > >                                             121 (Classless Static
> Route)
> > >                                              15 (Domainname)
> > >                                               6 (DNS server)
> > >                                              12 (Host name)
> > >                                              40 (NIS domain)
> > >                                              41 (NIS servers)
> > >                                              42 (NTP servers)
> > >                                              26 (Interface MTU)
> > >                                             119 (Domain Search)
> > >                                               3 (Routers)
> > >                                             121 (Classless Static
> Route)
> > >                                             249 (MSFT - Classless
> route)
> > >                                              42 (NTP servers)
> > >
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > >   TIME: 2013-11-18 08:24:53.998
> > >     IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> > >     OP: 1 (BOOTPREQUEST)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 060d4c79
> > >   SECS: 7
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 0.0.0.0
> > > SIADDR: 0.0.0.0
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> > > OPTION:  50 (  4) Request IP address        10.97.38.116
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> > >                                              28 (Broadcast address)
> > >                                               2 (Time offset)
> > >                                             121 (Classless Static
> Route)
> > >                                              15 (Domainname)
> > >                                               6 (DNS server)
> > >                                              12 (Host name)
> > >                                              40 (NIS domain)
> > >                                              41 (NIS servers)
> > >                                              42 (NTP servers)
> > >                                              26 (Interface MTU)
> > >                                             119 (Domain Search)
> > >                                               3 (Routers)
> > >                                             121 (Classless Static
> Route)
> > >                                             249 (MSFT - Classless
> route)
> > >                                              42 (NTP servers)
> > >
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > >   TIME: 2013-11-18 08:25:02.003
> > >     IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> > >     OP: 1 (BOOTPREQUEST)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 91cdcd74
> > >   SECS: 0
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 0.0.0.0
> > > SIADDR: 0.0.0.0
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         1 (DHCPDISCOVER)
> > > OPTION:  50 (  4) Request IP address        10.97.38.116
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> > >                                              28 (Broadcast address)
> > >                                               2 (Time offset)
> > >                                             121 (Classless Static
> Route)
> > >                                              15 (Domainname)
> > >                                               6 (DNS server)
> > >                                              12 (Host name)
> > >                                              40 (NIS domain)
> > >                                              41 (NIS servers)
> > >                                              42 (NTP servers)
> > >                                              26 (Interface MTU)
> > >                                             119 (Domain Search)
> > >                                               3 (Routers)
> > >                                             121 (Classless Static
> Route)
> > >                                             249 (MSFT - Classless
> route)
> > >                                              42 (NTP servers)
> > >
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > >   TIME: 2013-11-18 08:25:02.014
> > >     IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
> > >     OP: 2 (BOOTPREPLY)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 91cdcd74
> > >   SECS: 0
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 10.97.38.116
> > > SIADDR: 10.97.38.113
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         2 (DHCPOFFER)
> > > OPTION:  54 (  4) Server identifier         10.97.38.113
> > > OPTION:  51 (  4) IP address leasetime      -1 ()
> > > OPTION:   1 (  4) Subnet mask               255.255.255.0
> > > OPTION:  28 (  4) Broadcast address         10.97.38.255
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
> > > OPTION:   3 (  4) Routers                   10.97.38.1
> > > OPTION:  15 ( 17) Domainname                cs1cloud.internal
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > >   TIME: 2013-11-18 08:25:02.015
> > >     IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
> > >     OP: 1 (BOOTPREQUEST)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 91cdcd74
> > >   SECS: 0
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 0.0.0.0
> > > SIADDR: 0.0.0.0
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
> > > OPTION:  54 (  4) Server identifier         10.97.38.113
> > > OPTION:  50 (  4) Request IP address        10.97.38.116
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:  55 ( 16) Parameter Request List      1 (Subnet mask)
> > >                                              28 (Broadcast address)
> > >                                               2 (Time offset)
> > >                                             121 (Classless Static
> Route)
> > >                                              15 (Domainname)
> > >                                               6 (DNS server)
> > >                                              12 (Host name)
> > >                                              40 (NIS domain)
> > >                                              41 (NIS servers)
> > >                                              42 (NTP servers)
> > >                                              26 (Interface MTU)
> > >                                             119 (Domain Search)
> > >                                               3 (Routers)
> > >                                             121 (Classless Static
> Route)
> > >                                             249 (MSFT - Classless
> route)
> > >                                              42 (NTP servers)
> > >
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > >   TIME: 2013-11-18 08:25:02.016
> > >     IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
> > >     OP: 2 (BOOTPREPLY)
> > >  HTYPE: 1 (Ethernet)
> > >   HLEN: 6
> > >   HOPS: 0
> > >    XID: 91cdcd74
> > >   SECS: 0
> > >  FLAGS: 0
> > > CIADDR: 0.0.0.0
> > > YIADDR: 10.97.38.116
> > > SIADDR: 10.97.38.113
> > > GIADDR: 0.0.0.0
> > > CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
> > >  SNAME: .
> > >  FNAME: .
> > > OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
> > > OPTION:  54 (  4) Server identifier         10.97.38.113
> > > OPTION:  51 (  4) IP address leasetime      -1 ()
> > > OPTION:   1 (  4) Subnet mask               255.255.255.0
> > > OPTION:  28 (  4) Broadcast address         10.97.38.255
> > > OPTION:  12 ( 16) Host name                 centos6-template
> > > OPTION:   6 (  8) DNS server                10.97.38.113,10.97.32.20
> > > OPTION:   3 (  4) Routers                   10.97.38.1
> > > OPTION:  15 ( 17) Domainname                cs1cloud.internal
> > >
> >
> ---------------------------------------------------------------------------
> > >
> > > root@r-21-VM:~#
> > >
> > >
> > > Best Regards,
> > >
> > >
> > >
> > > Adam Scarcella
> > >
> > >
> > > On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <david@gnsa.us> wrote:
> > >
> > >> So having seen this specific problem scores of times, it's almost
> > >> always network config related. Several potential problems to look out
> > >> for:
> > >>
> > >> 1. Another DHCP server on the same network. (There are ways of
> > >> mitigating this, but for the time being, lets just say that
> > >> CloudStack's VR should be the only DHCP server on the network.
> > >>
> > >> 2. Shell into the physical machine that the VR is running on. Using
> > >> tcpdump (with filters of course) Do you see the broadcast asking for a
> > >> DHCP address coming from the VM on the other physical machine? Do you
> > >> see the VR answering the request? If so, move to step 3. If not (for
> > >> either of those questions) - you aren't communicating between the two
> > >> physical hosts - your switch config is suspect.
> > >>
> > >> 3. Shell into the physical machine that the VM is running on - You've
> > >> seen the response with an IP address assigned go out - so see if it is
> > >> seen by the physical host NIC? If not, you've again. If you see the
> > >> response (and are sure it's from the VR and not another DHCP server)
> > >> come back and tell us - something funky is going on.
> > >>
> > >> --David
> > >>
> > >> On Sun, Nov 17, 2013 at 10:11 AM, Adam <adam.scarcella@gmail.com>
> > wrote:
> > >> > Yes, I've disabled the firewall on both KVM hosts in question and
> > still
> > >> no
> > >> > dice. I can't even ping the VR from my guest VM, but when I set the
> > eth0
> > >> > device on the guest to static everything works fine, which makes no
> > >> sense
> > >> > at all to me. Simply setting the NIC to static allows me see the VR.
> > >> > Switching back to DHCP kills it again. I do not understand what is
> > >> required
> > >> > for the guests to acquire a DHCP lease from the VR. I know that the
> VR
> > >> is
> > >> > running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more
> > info,
> > >> > but only see a DHCP request when the guest is on the same KVM host
> as
> > >> the
> > >> > VR.
> > >> >
> > >> > Does anyone know exactly how to troubleshoot this scenario? I'm not
> > even
> > >> > sure what to look for.
> > >> >
> > >> > -Adam
> > >> >
> > >> > Best Regards,
> > >> >
> > >> >
> > >> >
> > >> > Adam Scarcella
> > >> >
> > >> >
> > >> > On Sat, Nov 16, 2013 at 5:43 PM, Carlos ReƔtegui <
> creategui@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Did you also check the host firewall?  Try disabling.
> > >> >>
> > >> >> > On Nov 16, 2013, at 1:51 PM, Adam <adam.scarcella@gmail.com>
> > wrote:
> > >> >> >
> > >> >> > The hosts have static IPs (10.97.38.[10-14]) and can all
see and
> > >> talk to
> > >> >> > each other via IP and hostname. I'm only using a basic zone
so no
> > >> VLAN
> > >> >> > tagging or anything funky like that.
> > >> >> >
> > >> >> > Best Regards,
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > Adam Scarcella
> > >> >> >
> > >> >> >
> > >> >> > On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky <
> > >> andrei@arhont.com
> > >> >> >wrote:
> > >> >> >
> > >> >> >>
> > >> >> >>
> > >> >> >> Adam, it sounds like a networking issue, check that all
hosts
> can
> > >> talk
> > >> >> to
> > >> >> >> each other on the same vlan that is used for guest network.
I
> had
> > >> the
> > >> >> same
> > >> >> >> issue with advanced networking when my vlans were not
properly
> > >> setup on
> > >> >> the
> > >> >> >> switches.
> > >> >> >>
> > >> >> >> Andrei
> > >> >> >>
> > >> >> >> ----- Original Message -----
> > >> >> >>
> > >> >> >> From: "Adam" <adam.scarcella@gmail.com>
> > >> >> >> To: users@cloudstack.apache.org
> > >> >> >> Sent: Saturday, 16 November, 2013 7:08:50 PM
> > >> >> >> Subject: Guest VMs not able to acquire DHCP IP from Virtual
> Router
> > >> >> unless
> > >> >> >> Guest and VR are on the same KVM host
> > >> >> >>
> > >> >> >> Hi All,
> > >> >> >>
> > >> >> >> I have a new and very strange issue that for the life
of me I
> > cannot
> > >> >> seem
> > >> >> >> to track down and fix.
> > >> >> >>
> > >> >> >> I have CS 4.2 running on 5 hosts in a simple basic zone.
All is
> > >> working
> > >> >> >> fine, except that my Guest VMs cannot seem to get a DHCP
lease
> > from
> > >> the
> > >> >> >> Virtual Router unless I migrate the Guest VM to the same
> physical
> > >> KVM
> > >> >> >> Host running the VR. That would seem to indicate a firewall
> issue,
> > >> but
> > >> >> I've
> > >> >> >> tested by turning off both the firewalls (VR KVM Host
iptables &
> > >> Guest
> > >> >> VM
> > >> >> >> KVM Host iptables). It didn't help. The only way to fix
it is to
> > >> migrate
> > >> >> >> the Guest VM to the same KVM Host that's running the
Virtual
> > Router.
> > >> >> >>
> > >> >> >> NOTE: The Console Proxy has worked flawlessly this whole
time.
> > >> >> >>
> > >> >> >> So, if a Guest VM starts on a different physical KVM
Host, it
> will
> > >> not
> > >> >> get
> > >> >> >> an internal IP of 169.254.x.x. All along the Console
Proxy works
> > >> fine.
> > >> >> Then
> > >> >> >> if I migrate the Guest VM to the same KVM host that's
running
> the
> > >> VR,
> > >> >> DHCP
> > >> >> >> automatically starts working and the Guest VM receives
a proper
> IP
> > >> >> address
> > >> >> >> of 10.97.38.x. Then I can migrate the Guest VM back to
any other
> > >> >> physical
> > >> >> >> KVM Host and believe it or not, it continues to work
flawlessly,
> > >> until I
> > >> >> >> either reboot the VM or restart the network services.
Then it
> > >> cannot see
> > >> >> >> the VR again and instead receives an internal IP of 169.254.x.x.
> > If
> > >> I
> > >> >> set a
> > >> >> >> static IP address & DNS everything works fine, no
matter where
> the
> > >> >> Guest VM
> > >> >> >> is running.
> > >> >> >>
> > >> >> >> Setting static IPs is not an option
> > >> >> >> Running all the Guest VMs on the same physical KVM host
is not
> an
> > >> option
> > >> >> >>
> > >> >> >> I desperately need to track down the root cause of this
issue
> so I
> > >> can
> > >> >> >> release this cloud to my entire department by Monday
morning.
> > >> Someone
> > >> >> >> please help!
> > >> >> >>
> > >> >> >> Best Regards,
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> Adam Scarcella
> > >> >> >>
> > >> >> >>
> > >> >>
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message