cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Miller <Evan.Mil...@citrix.com>
Subject RE: [SOLVED] Cannot add vm to any host in any cluster - error: "Unable to apply dhcp entry on router" - still completely blocked
Date Fri, 27 Jul 2012 03:32:43 GMT
Hi:

Not being able to add a VM to any host in a 3.0.2.1 Basic Network
Setup was caused by:

dhcp error(s) encountered by the system vm

These error(s) occurred because there was an unknown DHCP Server
on the same subnet as the system vm, but that unknown DHCP Server
was not included in the CSMS IP Address Space for management devices.
When we logged into the system vm, we saw that eth0 was associated
with the ip address of this (rogue and out-of-bounds) DHCP Server.
After we shutdown this DHCP Server, then attempted to deploy a VM again,
CSMS was able to go to the correct DHCP server and add the VM.

Sometimes, these surprises happen in a loosely controlled, multi-user 
engineering lab environment. :-|  :-)

Big Thanks to Ahmad for debugging the problem! Yeah!

Regards,
Evan

-----Original Message-----
From: Evan Miller [mailto:Evan.Miller@citrix.com] 
Sent: Wednesday, July 25, 2012 8:57 PM
To: cloudstack-users@incubator.apache.org
Subject: RE: Cannot add vm to any host in any cluster - error: "Unable to apply dhcp entry
on router" - still completely blocked

Abhinandan:

Attaching the management log for one of the failed attempts, earlier today ...

Do you need other information?

Evan

-----Original Message-----
From: Evan Miller
Sent: Wednesday, July 25, 2012 8:38 PM
To: cloudstack-users@incubator.apache.org
Subject: RE: Cannot add vm to any host in any cluster - error: "Unable to apply dhcp entry
on router" - still completely blocked

I can ssh without error to the system vm from only one of the XenServers that is having trouble
 adding VMs. That particular XenServer was setup as part of the CSMS Basic Network Setup.
Here are the good results:

[root@xenserver1-cs .ssh]# ssh -p 3922 -i /root/.ssh/id_rsa.cloud 169.254.0.227 Linux v-2-VM
2.6.32-5-686-bigmem #1 SMP Mon Jan 16 16:42:05 UTC 2012 i686

The programs included with the Debian GNU/Linux system are free software; the exact distribution
terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable
law.
Last login: Thu Jul 26 03:27:37 2012 from 169.254.0.1 Linux v-2-VM 2.6.32-5-686-bigmem #1
SMP Mon Jan 16 16:42:05 UTC 2012 i686

The programs included with the Debian GNU/Linux system are free software; the exact distribution
terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable
law.
root@v-2-VM:~#

AND

Here are the bad results of the XenServer Host that was added after CSMS Basic Network Setup:

[root@xenserver03 .ssh]# ssh -p 3922 -i /root/.ssh/id_rsa.cloud 169.254.0.227
ssh: connect to host 169.254.0.227 port 3922: No route to host
[root@xenserver03 .ssh]#

For the problematic xenserver03, its known_hosts file is empty.
While, for the successful xenserver1-cs, its known_hosts file contains entries for:
   CSMS Server,
   two 169.254.*.* ip addresses.


Regards,
Evan



-----Original Message-----
From: Abhinandan Prateek [mailto:Abhinandan.Prateek@citrix.com]
Sent: Wednesday, July 25, 2012 7:56 PM
To: cloudstack-users@incubator.apache.org
Subject: RE: Cannot add vm to any host in any cluster - error: "Unable to apply dhcp entry
on router" - still completely blocked

Evan,

   There was a type in the command:
		ssh -p 3922 -i /root/.ssh/id_rsa.cloud  <rvm link local ip>    # a i not an I, I guess
outlook capitalized the i and got overlooked.
You need to run this on the host (hypervisor) where CS created the RVM for the account that
is giving you trouble. This is just to ensure that the path taken by CS to pass control commands(dhcp
is one of them) to RVM is good.

-abhi


>-----Original Message-----
>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>Sent: Thursday, July 26, 2012 5:15 AM
>To: cloudstack-users@incubator.apache.org
>Subject: RE: Cannot add vm to any host in any cluster - error: "Unable 
>to apply dhcp entry on router" - still completely blocked
>
>Abhinandan:
>
>I don't understand which ip and which host you are talking about in 
>your
>comment:
>
>To check try logging into RVM using ssh keys from the host on which it 
>resides (ssh -p 3922 -I /root/.ssh/id_rsa.cloud  <rvm link local ip>).
>
>At any rate, I tried this from the XenServer that is unable to add the VM:
>
>[root@xenserver1-cs .ssh]# ssh -p 3922 -I /root/.ssh/id_rsa.cloud
>169.254.0.227 no support for smartcards.
>The authenticity of host '169.254.0.227 (169.254.0.227)' can't be established.
>RSA key fingerprint is 58:de:8b:d7:2b:8e:01:2e:cd:f4:58:86:27:e4:94:39.
>Are you sure you want to continue connecting (yes/no)? yes
>Warning: Permanently added '169.254.0.227' (RSA) to the list of known hosts.
>Permission denied (publickey).
>[root@xenserver1-cs .ssh]#
>
>I assumed the RVM's ip address is the Link Local Address (169.254.0.227)?
>If not correct, please advise.
>
>What next?
>
>Note:
>Before trying the above, I uninstalled CSMS 3.0.2.1 and (part of?) the 
>database using install.sh. Then, I re-ran the Basic Network Setup from 
>the GUI. The above DHCP problem still persists.
>
>Please help. I am still completely blocked.
>
>Evan
>
>-----Original Message-----
>From: Abhinandan Prateek [mailto:Abhinandan.Prateek@citrix.com]
>Sent: Monday, July 23, 2012 9:11 PM
>To: cloudstack-users@incubator.apache.org
>Subject: RE: Cannot add vm to any host in any cluster - error: "Unable 
>to apply dhcp entry on router"
>
>Another issue can be that the ssh keys are not programmed correctly 
>either on the host or the routerVm, due to which the dhcp script fails.
>To check try logging into RVM using ssh keys from the host on which it 
>resides (ssh -p 3922 -I /root/.ssh/id_rsa.cloud  <rvm link local ip>).
>If this fails, check the configuration table for ssh.publickey and ssh.privatekey.
>
>-abhi
>
>>-----Original Message-----
>>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>>Sent: Tuesday, July 24, 2012 8:29 AM
>>To: cloudstack-users@incubator.apache.org
>>Subject: RE: Cannot add vm to any host in any cluster - error: "Unable 
>>to apply dhcp entry on router"
>>
>>Can someone please answer follow-up questions from early this morning?
>>
>>My CSMS Server is not adding any VMs.
>>I am *completely* blocked by:
>>
>>Unable to apply dhcp entry on router
>>
>>
>>Regards,
>>Evan
>>
>>-----Original Message-----
>>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>>Sent: Monday, July 23, 2012 5:48 AM
>>To: cloudstack-users@incubator.apache.org
>>Subject: RE: Cannot add vm to any host in any cluster - error: "Unable 
>>to apply dhcp entry on router"
>>
>>How to do all of those router changes from the GUI?
>>If not from the GUI, which API commands should be executed?
>>More details would be helpful.
>>
>>Regards,
>>Evan
>>
>>
>>
>>-----Original Message-----
>>From: Rajesh Battala [mailto:rajesh.battala@citrix.com]
>>Sent: Sunday, July 22, 2012 7:06 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: RE: Cannot add vm to any host in any cluster - error: "Unable 
>>to apply dhcp entry on router"
>>
>>Evan,
>>Did you check the status of the VR?. Is DHCP service is running in the VR?
>>Please check those services.
>>Sometimes I had observed DHCP service will fail to start on a particular eth.
>>Quick workaround is restart VR , check the DHCP service and then try 
>>to create the instances.
>>
>>Thanks
>>Rajesh Battala
>>
>>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>>Sent: Sunday, July 22, 2012 12:51 AM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Cannot add vm to any host in any cluster - error: "Unable to 
>>apply dhcp entry on router"
>>
>>Running CloudStack Management Server:
>>  v3.0.2.1
>>On:
>>  [root@cumulus management]# uname -a
>>  Linux cumulus.eng.citrite.net 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
>>19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>  [root@cumulus management]#
>>Hypervisor :
>>  XenServer v6.02
>>
>>Hi:
>>
>>I can't add any VMs to any hosts in my 3 cluster, two host cloud.
>>
>>In one attempt, the GUI basically says:
>>
>>   Unable to create a deployment for VM[User|i-2-7-VM]
>>
>>Here is small part of the attached management log snippet.
>>
>>2012-07-21 10:49:25,568 DEBUG [agent.manager.AgentAttache]
>>(DirectAgent-
>>62:null) Seq 1-564527128: No more commands found
>>2012-07-21 10:49:25,568 INFO  [cloud.vm.VirtualMachineManagerImpl]
>>(Job-
>>Executor-1:job-11) Unable to contact resource.
>>com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is
>>unreachable: Unable to apply dhcp entry on router
>>        at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRule
>s
>>(VirtualNetworkApplianceManagerImpl.java:2802)
>>        at
>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhc
>p
>>Entry(VirtualNetworkApplianceManagerImpl.java:2089)
>>        at
>>com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRo
>ut
>>erElement.java:738)
>>        at
>>com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManag
>e
>>rImpl.java:1792)
>>        at
>>com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.ja
>v
>>a:1887)
>>        at
>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMa
>n
>>agerImpl.java:741)
>>        at
>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm
>p
>>l.java:461)
>>        at
>>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerI
>m
>>pl.java:2580)
>>        at
>>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerI
>m
>>pl.java:2517)
>>        at
>>com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java
>>:32
>)
>>        at
>>com.cloud.api.commands.DeployVMCmd.execute(DeployVMCmd.java:317)
>>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
>>        at
>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:4
>2
>>7)
>>        at
>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>        at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>a
>>va
>>:1110)
>>        at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>j
>>av
>>a:603)
>>        at java.lang.Thread.run(Thread.java:679)
>>
>>How should this be fixed?
>>Can the problem be fixed from the GUI?
>>Please advise.
>>
>>Regards,
>>
>>Evan Miller
>>
>>Citrix Systems. Inc.
>>Desktop and Cloud Engineering Infrastructure
>>4988 Great America Parkway
>>Santa Clara, CA 95054


Mime
View raw message