cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Lukzak <mis...@vp.pl>
Subject Re[4]: Urgent help needed. Can't start Virtual Router.
Date Thu, 07 Aug 2014 19:43:06 GMT
I removed this host from CS, but still problem is.

How I can "rebuild VR" from zero?
It's looks like CS want to start VM (witch really is a VirtualRouter) that not exists on the
host.

INFO  [xen.resource.CitrixResourceBase] (DirectAgent-148:) VM does not exist on XenServer11681b28-5f38-4f20-aaae-0c9ed1e9ecc7

Or other way. How I can bind new network to VM?
Ony way is modify VM parameter in database?

Michael





> depending on your setup, but what I would do move all your Instances from
> host machine where the VR lives and reboot. Again, that's what I would do.

> you can try this link for XenServer 5
> http://blogs.citrix.com/2009/01/26/how-to-configure-an-nfs-based-iso-sr-using-local-storage-on-a-xenserver5-host/

> Thanks,


> On Thu, Aug 7, 2014 at 12:17 PM, Michael Lukzak <miszko@vp.pl> wrote:

>> Hi,

>> We are usning xenserver from Citrix.

>> I found some clue. In console to server on this host I see that nfs has
>> error "Stale NFS file handle".

>> How to remount NFS? Maybe this is a problem?
>> But why other VR are started and few of them not... ;(

>> Best regards,
>> Michael



>> > Hi Michael,
>> > if it helps, I would restart the agent on the host machine where VR
>> lives.

>> > it depends on hypervisor but on KVM, following command: service
>> > cloudstack-agent restart


>> > Thanks,
>> > Motty


>> > On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <miszko@vp.pl> wrote:

>> >> Hi,

>> >> One of our host are died (after very high load).
>> >> Host has a lot of Virtual Routers running.

>> >> We are using advanced networking.

>> >> Only way to bring machine alive was hard reset.

>> >> After this CloudStack decied to start all of VirtualRouterts,
>> >> when server was back online.

>> >> Most of VR are started, but couple don't.
>> >> Cloudstack want to start this VM, but can't ;(
>> >> Here are log from this action.

>> >> It's looks like that CS are looking for VR host on Xen host, but he
>> >> couldn't find.

>> >> Please help how to repair this situation.
>> >> Restart network with clean up options ending with the same resoult.

>> >> Is there any option to recreate VR?
>> >> I tried to remove VR from Infrastructure -> Virtual Router, but there
is
>> >> still problem
>> >> with create new one.



>> >> Here are logs from "restart network with clean up options"
>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated
>> >> with any VPC
>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated
>> >> with any VPC
>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not
>> associated
>> >> with any VPC
>> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526)
>> >> Failed to start instance VM[DomainRouter|r-1015-VM]
>> >> java.lang.NullPointerException
>> >>         at
>> >>
>> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>> >>         at
>> >>
>> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
>> >>         at
>> >>
>> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
>> >>         at
>> >>
>> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473)
>> >>         at
>> >>
>> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381)
>> >>         at
>> >>
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> >>         at
>> >>
>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92)
>> >>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>> >>         at
>> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>> >>         at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>         at
>> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>         at java.lang.Thread.run(Thread.java:722)
>> >> INFO  [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not
>> >> exist on XenServer0d1d5015-4061-4554-96de-406155449296
>> >> WARN  [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526)
>> >> Failed to implement network Ntwk[338|Guest|8] elements and resources as
>> a
>> >> part of network restart due to
>> >> com.cloud.exception.AgentUnavailableException: Resource [Host:9] is
>> >> unreachable: Host 9: Unable to start instance due to null
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>> >>         at
>> >>
>> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
>> >>         at
>> >>
>> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
>> >>         at
>> >>
>> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473)
>> >>         at
>> >>
>> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381)
>> >>         at
>> >>
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> >>         at
>> >>
>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92)
>> >>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>> >>         at
>> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>> >>         at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>         at
>> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>         at java.lang.Thread.run(Thread.java:722)
>> >> Caused by: java.lang.NullPointerException
>> >>         at
>> >>
>> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>> >>         ... 23 more
>> >> WARN  [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526)
>> >> Network id=338 failed to restart.
>> >> WARN  [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526)
>> Unable
>> >> to unregister active job 6526 from JMX monitoring


>> >> Here are log when I want to start VR (his state is stopeed, I want to
>> >> start and got this)

>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated
>> >> with any VPC
>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated
>> >> with any VPC
>> >> WARN  [network.element.VpcVirtualRouterElement]
>> >> (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not
>> associated
>> >> with any VPC
>> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528)
>> >> Failed to start instance VM[DomainRouter|r-1015-VM]
>> >> java.lang.NullPointerException
>> >>         at
>> >>
>> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>> >>         at
>> >>
>> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>> >>         at
>> >>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874)
>> >>         at
>> >>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824)
>> >>         at
>> >>
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> >>         at
>> >>
>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103)
>> >>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>> >>         at
>> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>> >>         at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>         at
>> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>         at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>         at java.lang.Thread.run(Thread.java:722)


>> >> Best Regards,
>> >> Michael



>> --
>> Pozdrowienia,
>>  Michael Lukzak



-- 
Pozdrowienia,
 Michael Lukzak
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message