cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nagaraju (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-8729) Unable to start a VM due to concurrent operation
Date Thu, 13 Aug 2015 09:32:46 GMT
Nagaraju created CLOUDSTACK-8729:
------------------------------------

             Summary: Unable to start a VM due to concurrent operation
                 Key: CLOUDSTACK-8729
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8729
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
            Reporter: Nagaraju


While creating Instance getting the Error  "Unable to start a VM due to concurrent operation"

below is the log details ,can any one help me in this ?









tal: 270766178304; new used: 2415919104, reserved: 0; requested mem: 536870912,alloc_from_last:false
2015-08-13 14:56:08,805 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) VM is being created in podId: 1
2015-08-13 14:56:08,809 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Network id=205 is already implemented
2015-08-13 14:56:08,813 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-9:ctx-67395b21 job-501/job-503
ctx-cd154467) Service SecurityGroup is not supported in the network id=205
2015-08-13 14:56:08,815 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Changing active number of nics for network id=205 on 1
2015-08-13 14:56:08,819 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Asking VirtualRouter to prepare for Nic[560-145-null-10.96.162.3]
2015-08-13 14:56:08,824 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Lock is acquired for network id 205 as a part of router startup
in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(145|ROOT-->Pool(1))]
2015-08-13 14:56:08,826 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Lock is released for network id 205 as a part of router startup
in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(145|ROOT-->Pool(1))]
2015-08-13 14:56:08,830 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-9:ctx-67395b21 job-501/job-503
ctx-cd154467) Service SecurityGroup is not supported in the network id=205
2015-08-13 14:56:08,836 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-9:ctx-67395b21 job-501/job-503
ctx-cd154467) Service SecurityGroup is not supported in the network id=205
2015-08-13 14:56:08,839 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Applying dhcp entry in network Ntwk[205|Guest|7]
2015-08-13 14:56:08,845 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Router requires upgrade. Unable to send command to router:140,
router template version : Cloudstack Release 4.4.0 Wed Jul 30 15:11:52 UTC 2014, minimal required
version : 4.5.0
2015-08-13 14:56:08,846 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Failed to start instance VM[User|i-2-145-VM]
com.cloud.utils.exception.CloudRuntimeException: Unable to send command. Upgrade in progress.
Please contact administrator.
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3796)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3217)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4079)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:3209)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy191.applyDhcpEntry(Unknown Source)
        at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:921)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1245)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1367)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1303)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:986)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4480)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2015-08-13 14:56:08,848 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Cleaning up resources for the vm VM[User|i-2-145-VM] in Starting
state
2015-08-13 14:56:08,851 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Seq 1-4118260384253608023: Forwarding Seq 1-4118260384253608023:
 { Cmd , MgmtId: 52234440052, via: 1(cstkvm1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-145-VM","wait":0}}]
} to 52234911571
2015-08-13 14:56:08,971 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:null) Seq 1-4118260384253608023:
Processing:  { Ans: , MgmtId: 52234440052, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}]
}
2015-08-13 14:56:08,971 DEBUG [c.c.a.t.Request] (Work-Job-Executor-9:ctx-67395b21 job-501/job-503
ctx-cd154467) Seq 1-4118260384253608023: Received:  { Ans: , MgmtId: 52234440052, via: 1,
Ver: v1, Flags: 10, { StopAnswer } }
2015-08-13 14:56:08,975 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Changing active number of nics for network id=205 on -1
2015-08-13 14:56:08,978 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Successfully released network resources for the vm VM[User|i-2-145-VM]
2015-08-13 14:56:08,978 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Successfully cleanued up resources for the vm VM[User|i-2-145-VM]
in Starting state
2015-08-13 14:56:08,983 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) VM state transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 1
2015-08-13 14:56:08,986 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Hosts's actual total CPU: 92000 and CPU after applying overprovisioning:
92000
2015-08-13 14:56:08,986 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Hosts's actual total RAM: 270766170112 and RAM after applying
overprovisioning: 270766178304
2015-08-13 14:56:08,986 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) release cpu from host: 1, old used: 2000,reserved: 0, actual
total: 92000, total with overprovisioning: 92000; new used: 1500,reserved:0; movedfromreserved:
false,moveToReserveredfalse
2015-08-13 14:56:08,986 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) release mem from host: 1, old used: 2415919104,reserved: 0,
total: 270766178304; new used: 1879048192,reserved:0; movedfromreserved: false,moveToReserveredfalse
2015-08-13 14:56:08,991 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to send command.
Upgrade in progress. Please contact administrator.
2015-08-13 14:56:08,991 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503 ctx-cd154467) Rethrow exception com.cloud.exception.AgentUnavailableException:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to send command.
Upgrade in progress. Please contact administrator.
2015-08-13 14:56:08,992 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 145, job origin:
501
2015-08-13 14:56:08,992 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Unable to complete AsyncJobVO {id:503, userId: 2, accountId: 2, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAkXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
52234440052, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Aug 13
14:56:08 IST 2015}, job origin:501
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable
to start instance due to Unable to send command. Upgrade in progress. Please contact administrator.
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1109)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4480)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to send command. Upgrade
in progress. Please contact administrator.
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3796)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3217)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4079)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:3209)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy191.applyDhcpEntry(Unknown Source)
        at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:921)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1245)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1367)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1303)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:986)
        ... 20 more
2015-08-13 14:56:08,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Complete async job-503, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC1jb20uY2xvdWQuZXhjZXB0aW9uLkFnZW50VW5hdmFpbGFibGVFeGNlcHRpb24AAAAAVk1PdwIAAHhyADBjb20uY2xvdWQuZXhjZXB0aW9uLlJlc291cmNlVW5hdmFpbGFibGVFeGNlcHRpb24AAAAAVk1PfwIAAkoAA19pZEwABl9zY29wZXQAEUxqYXZhL2xhbmcvQ2xhc3M7eHIAImNvbS5jbG91ZC5leGNlcHRpb24uQ2xvdWRFeGNlcHRpb2556I4b8eRExwIAAkwAC2NzRXJyb3JDb2RldAATTGphdmEvbGFuZy9JbnRlZ2VyO0wABmlkTGlzdHQAFUxqYXZhL3V0aWwvQXJyYXlMaXN0O3hyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3IAL2NvbS5jbG91ZC51dGlscy5leGNlcHRpb24uQ2xvdWRSdW50aW1lRXhjZXB0aW9uAAAAAFZNT3IDAAFJAAtjc0Vycm9yQ29kZXhyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHEAfgAGcQB-AA90AEpVbmFibGUgdG8gc2VuZCBjb21tYW5kLiBVcGdyYWRlIGluIHByb2dyZXNzLiBQbGVhc2UgY29udGFjdCBhZG1pbmlzdHJhdG9yLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAKHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAJTAAIZmlsZU5hbWVxAH4ACUwACm1ldGhvZE5hbWVxAH4ACXhwAAAO1HQAO2NvbS5jbG91ZC5uZXR3b3JrLnJvdXRlci5WaXJ0dWFsTmV0d29ya0FwcGxpYW5jZU1hbmFnZXJJbXBsdAAnVmlydHVhbE5ldHdvcmtBcHBsaWFuY2VNYW5hZ2VySW1wbC5qYXZhdAAUc2VuZENvbW1hbmRzVG9Sb3V0ZXJzcQB-ABMAAAyRdAA9Y29tLmNsb3VkLm5ldHdvcmsucm91dGVyLlZpcnR1YWxOZXR3b3JrQXBwbGlhbmNlTWFuYWdlckltcGwkN3EAfgAWdAAHZXhlY3V0ZXNxAH4AEwAAD-9xAH4AFXEAfgAWdAAKYXBwbHlSdWxlc3NxAH4AEwAADIlxAH4AFXEAfgAWdAAOYXBwbHlEaGNwRW50cnlzcQB-ABP____-dAAkc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2Uwc3EAfgATAAAAOXEAfgAgcQB-ACF0AAZpbnZva2VzcQB-ABMAAAArdAAoc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAkc3EAfgATAAACXnQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACRzcQB-ABMAAAE9dAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25zcQB-ABMAAAC3dAA8b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLlJlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9udAAfUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb24uamF2YXQAD2ludm9rZUpvaW5wb2ludHNxAH4AEwAAAJZxAH4AMHEAfgAxdAAHcHJvY2VlZHNxAH4AEwAAAFt0AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAkc3EAfgATAAAArHEAfgAwcQB-ADFxAH4ANHNxAH4AEwAAAMx0ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AJHNxAH4AE_____90ABdjb20uc3VuLnByb3h5LiRQcm94eTE5MXBxAH4AHnNxAH4AEwAAA5l0AC5jb20uY2xvdWQubmV0d29yay5lbGVtZW50LlZpcnR1YWxSb3V0ZXJFbGVtZW50dAAZVmlydHVhbFJvdXRlckVsZW1lbnQuamF2YXQADGFkZERoY3BFbnRyeXNxAH4AEwAABN10AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uTmV0d29ya09yY2hlc3RyYXRvcnQAGE5ldHdvcmtPcmNoZXN0cmF0b3IuamF2YXQADnByZXBhcmVFbGVtZW50c3EAfgATAAAFV3EAfgBDcQB-AER0AApwcmVwYXJlTmljc3EAfgATAAAFF3EAfgBDcQB-AER0AAdwcmVwYXJlc3EAfgATAAAD2nQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHNxAH4AEwAAEYBxAH4AS3EAfgBMcQB-AE1zcQB-ABP____-cQB-ACBxAH4AIXEAfgAic3EAfgATAAAAOXEAfgAgcQB-ACFxAH4AJHNxAH4AEwAAACtxAH4AJnEAfgAncQB-ACRzcQB-ABMAAAJecQB-AClxAH4AKnEAfgAkc3EAfgATAAAAa3QAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnNxAH4AEwAAEhxxAH4AS3EAfgBMcQB-AFZzcQB-ABMAAABndAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-ABMAAAIZdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0c3EAfgATAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EAfgATAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNxAH4AEwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-AGZ0AA9jYWxsV2l0aENvbnRleHRzcQB-ABMAAAA1cQB-AGlxAH4AZnQADnJ1bldpdGhDb250ZXh0c3EAfgATAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgBicQB-AGNzcQB-ABMAAAHucQB-AF1xAH4AXnEAfgBjc3EAfgATAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgBnc3EAfgATAAABBnQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AY3NxAH4AEwAABHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4AEwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AHhxAH4AY3NxAH4AEwAAAul0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AY3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgALeHIALGphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAB3BAAAAAB4cQB-AIR4AAAQmncIAAAAAAAAAAB4dACUUmVzb3VyY2UgW0hvc3Q6MV0gaXMgdW5yZWFjaGFibGU6IEhvc3QgMTogVW5hYmxlIHRvIHN0YXJ0IGluc3RhbmNlIGR1ZSB0byBVbmFibGUgdG8gc2VuZCBjb21tYW5kLiBVcGdyYWRlIGluIHByb2dyZXNzLiBQbGVhc2UgY29udGFjdCBhZG1pbmlzdHJhdG9yLnVxAH4AEQAAABVzcQB-ABMAAARVcQB-AEtxAH4ATHEAfgBNc3EAfgATAAARgHEAfgBLcQB-AExxAH4ATXNxAH4AE_____5xAH4AIHEAfgAhcQB-ACJzcQB-ABMAAAA5cQB-ACBxAH4AIXEAfgAkc3EAfgATAAAAK3EAfgAmcQB-ACdxAH4AJHNxAH4AEwAAAl5xAH4AKXEAfgAqcQB-ACRzcQB-ABMAAABrcQB-AFRxAH4AVXEAfgBWc3EAfgATAAASHHEAfgBLcQB-AExxAH4AVnNxAH4AEwAAAGdxAH4AWXEAfgBacQB-AFtzcQB-ABMAAAIZcQB-AF1xAH4AXnEAfgBfc3EAfgATAAAAMXEAfgBhcQB-AGJxAH4AY3NxAH4AEwAAADhxAH4AZXEAfgBmcQB-AGdzcQB-ABMAAABncQB-AGlxAH4AZnEAfgBqc3EAfgATAAAANXEAfgBpcQB-AGZxAH4AbHNxAH4AEwAAAC5xAH4AbnEAfgBicQB-AGNzcQB-ABMAAAHucQB-AF1xAH4AXnEAfgBjc3EAfgATAAAB13EAfgBxcQB-AHJxAH4AZ3NxAH4AEwAAAQZxAH4AdHEAfgB1cQB-AGNzcQB-ABMAAAR5cQB-AHdxAH4AeHEAfgB5c3EAfgATAAACZ3EAfgB7cQB-AHhxAH4AY3NxAH4AEwAAAulxAH4AfXEAfgB-cQB-AGNxAH4AgnhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQvXNxAH4AgwAAAAB3BAAAAAB4AAAAAAAAAAF2cgATY29tLmNsb3VkLmhvc3QuSG9zdAJ4zp_gRYjJAgAAeHA
2015-08-13 14:56:08,995 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Publish async job-503 complete on message bus
2015-08-13 14:56:08,995 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Wake up jobs related to job-503
2015-08-13 14:56:08,995 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Update db status for job-503
2015-08-13 14:56:08,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Wake up jobs joined with job-503 and disjoin all subjobs created from job-
503
2015-08-13 14:56:08,999 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Done executing com.cloud.vm.VmWorkStart for job-503
2015-08-13 14:56:09,000 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-9:ctx-67395b21
job-501/job-503) Remove job-503 from job monitoring
2015-08-13 14:56:09,010 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501 ctx-2c34cfbb) Destroying vm VM[User|i-2-145-VM] as it failed to create on Host with
Id:null
2015-08-13 14:56:09,013 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501 ctx-2c34cfbb) VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's
original host id: null new host id: null host id before state transition: null
2015-08-13 14:56:09,030 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501 ctx-2c34cfbb) Access granted to Acct[7ea6ad12-40d0-11e5-8a7b-000c29860c0d-admin] to
Domain:1/ by AffinityGroupAccessChecker
2015-08-13 14:56:09,034 WARN  [o.a.c.alerts] (API-Job-Executor-18:ctx-f52cd315 job-501 ctx-2c34cfbb)
 alertType:: 8 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Failed to
deploy Vm with Id: 145, on Host with Id: null
2015-08-13 14:56:09,048 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-18:ctx-f52cd315
job-501) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:618)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:242)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3584)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3227)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy213.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin.execute(DeployVMCmdByAdmin.java:50)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable:
Host 1: Unable to start instance due to Unable to send command. Upgrade in progress. Please
contact administrator.
        ... 35 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to send command. Upgrade
in progress. Please contact administrator.
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3796)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3217)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4079)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:3209)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy191.applyDhcpEntry(Unknown Source)
        at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:921)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1245)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1367)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1303)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:986)
        ... 20 more
2015-08-13 14:56:09,050 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Complete async job-501, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to start a VM due to concurrent operation"}
2015-08-13 14:56:09,050 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Publish async job-501 complete on message bus
2015-08-13 14:56:09,050 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Wake up jobs related to job-501
2015-08-13 14:56:09,050 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Update db status for job-501
2015-08-13 14:56:09,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Wake up jobs joined with job-501 and disjoin all subjobs created from job- 501
2015-08-13 14:56:09,053 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-18:ctx-f52cd315
job-501) Done executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin for
job-501
2015-08-13 14:56:09,053 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-18:ctx-f52cd315
job-501) Remove job-501 from job monitoring
2015-08-13 14:56:10,575 DEBUG [o.a.c.s.RemoteHostEndPoint] (Timer-10:ctx-521fcd37) Sending
command org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3
2015-08-13 14:56:10,576 DEBUG [c.c.a.t.Request] (Timer-10:ctx-521fcd37) Seq 3-6006676003005399179:
Sending  { Cmd , MgmtId: 52234440052, via: 3(s-138-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","request":"GET_STATUS","hvm":false,"description":"CentOS
5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.204.206.10/cst/secondary","_role":"Image"}},"url":"http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://10.204.206.10/cst/secondary","wait":0}}]
}
2015-08-13 14:56:10,620 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) Seq 3-6006676003005399179:
Processing:  { Ans: , MgmtId: 52234440052, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","downloadPct":34,"errorString":"
","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1ccfebac-b23f-3d50-9904-7722a418ece1/template/tmpl/1/4/dnld4011508956757342167tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"
","wait":0}}] }
2015-08-13 14:56:10,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d3d28771)
Begin cleanup expired async-jobs
2015-08-13 14:56:10,736 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d3d28771)
End cleanup expired async-jobs
2015-08-13 14:56:10,849 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-99ff21f6)
Found 0 routers to update status. 
2015-08-13 14:56:10,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-99ff21f6)
Found 0 networks to update RvR status. 
2015-08-13 14:56:11,082 DEBUG [c.c.a.ApiServlet] (catalina-exec-10:ctx-34c806e4) ===START===
 10.204.207.174 -- GET  command=queryAsyncJobResult&jobId=3259941c-7893-4ff4-8108-28cddc101f0f&response=json&sessionkey=6XbZtu7CrIwY3L6MFvwsphx%2FLMY%3D&_=1439457972437
2015-08-13 14:56:11,092 DEBUG [c.c.a.ApiServlet] (catalina-exec-10:ctx-34c806e4 ctx-caf951c9)
===END===  10.204.207.174 -- GET  command=queryAsyncJobResult&jobId=3259941c-7893-4ff4-8108-28cddc101f0f&response=json&sessionkey=6XbZtu7CrIwY3L6MFvwsphx%2FLMY%3D&_=1439457972437
2015-08-13 14:56:11,107 DEBUG [c.c.a.ApiServlet] (catalina-exec-18:ctx-b3d122cb) ===START===
 10.204.207.174 -- GET  command=listVirtualMachines&id=f4e907f9-0587-4353-bd89-347567198eb2&response=json&sessionkey=6XbZtu7CrIwY3L6MFvwsphx%2FLMY%3D&_=1439457972458
2015-08-13 14:56:11,123 DEBUG [c.c.a.ApiServlet] (catalina-exec-18:ctx-b3d122cb ctx-e43ff29a)
===END===  10.204.207.174 -- GET  command=listVirtualMachines&id=f4e907f9-0587-4353-bd89-347567198eb2&response=json&sessionkey=6XbZtu7CrIwY3L6MFvwsphx%2FLMY%3D&_=1439457972458
2015-08-13 14:56:20,620 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-4:ctx-542d7efe)
Performing image store createTemplate async callback
2015-08-13 14:56:20,625 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-4:ctx-542d7efe)
Performing image store createTemplate async callback
2015-08-13 14:56:20,734 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cfd0a0a6)
Begin cleanup expired async-jobs
2015-08-13 14:56:20,738 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cfd0a0a6)
End cleanup expired async-jobs
2015-08-13 14:56:24,145 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-2c5c5474)
Zone 1 is ready to launch secondary storage VM
2015-08-13 14:56:24,365 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-08f39e6f)
Zone 1 is ready to launch console proxy
2015-08-13 14:56:25,856 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-59ddc04a) AutoScaling
Monitor is running...
2015-08-13 14:56:25,870 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-afb9b8b9) VmStatsCollector
is running...
2015-08-13 14:56:30,430 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-e91555b7) StorageCollector
is running...
2015-08-13 14:56:30,481 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e91555b7) Seq 3-6006676003005399180:
Received:  { Ans: , MgmtId: 52234440052, via: 3, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2015-08-13 14:56:30,484 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-e91555b7)
Seq 1-4118260384253608024: Forwarding null to 52234911571
2015-08-13 14:56:30,605 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e91555b7) Seq 1-4118260384253608024:
Received:  { Ans: , MgmtId: 52234440052, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
2015-08-13 14:56:30,620 DEBUG [o.a.c.s.RemoteHostEndPoint] (Timer-10:ctx-7190a724) Sending
command org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3
2015-08-13 14:56:30,621 DEBUG [c.c.a.t.Request] (Timer-10:ctx-7190a724) Seq 3-6006676003005399181:
Sending  { Cmd , MgmtId: 52234440052, via: 3(s-138-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","request":"GET_STATUS","hvm":false,"description":"CentOS
5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.204.206.10/cst/secondary","_role":"Image"}},"url":"http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://10.204.206.10/cst/secondary","wait":0}}]
}
2015-08-13 14:56:30,664 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) Seq 3-6006676003005399181:
Processing:  { Ans: , MgmtId: 52234440052, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","downloadPct":34,"errorString":"
","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1ccfebac-b23f-3d50-9904-7722a418ece1/template/tmpl/1/4/dnld4011508956757342167tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"
","wait":0}}] }
2015-08-13 14:56:30,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d2032270)
Begin cleanup expired async-jobs
2015-08-13 14:56:30,735 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d2032270)
End cleanup expired async-jobs
2015-08-13 14:56:31,899 DEBUG [c.c.s.d.DownloadListener] (Timer-10:ctx-0eca9eb1) Scheduling
timeout at 30000 ms, TEMPLATE: 4 at host 3
2015-08-13 14:56:33,043 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null)
Seq 3-7133983284731576352: Routing from 52234911571
2015-08-13 14:56:33,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null)
Seq 3-7133983284731576352: MgmtId 52234911571: Resp: Routing to peer
2015-08-13 14:56:40,664 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-8:ctx-30d1f19b)
Performing image store createTemplate async callback
2015-08-13 14:56:40,670 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-8:ctx-30d1f19b)
Performing image store createTemplate async callback
2015-08-13 14:56:40,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2dcc8371)
Begin cleanup expired async-jobs
2015-08-13 14:56:40,737 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2dcc8371)
End cleanup expired async-jobs
2015-08-13 14:56:40,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-859af02a)
Found 0 routers to update status. 
2015-08-13 14:56:40,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-859af02a)
Found 0 networks to update RvR status. 
2015-08-13 14:56:47,100 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-a0c735bc) HostStatsCollector
is running...
2015-08-13 14:56:47,105 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:ctx-a0c735bc)
Seq 1-4118260384253608025: Forwarding null to 52234911571
2015-08-13 14:56:47,718 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-a0c735bc) Seq 1-4118260384253608025:
Received:  { Ans: , MgmtId: 52234440052, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer
} }
2015-08-13 14:56:50,664 DEBUG [o.a.c.s.RemoteHostEndPoint] (Timer-10:ctx-a8f3e757) Sending
command org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3
2015-08-13 14:56:50,665 DEBUG [c.c.a.t.Request] (Timer-10:ctx-a8f3e757) Seq 3-6006676003005399182:
Sending  { Cmd , MgmtId: 52234440052, via: 3(s-138-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","request":"GET_STATUS","hvm":false,"description":"CentOS
5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.204.206.10/cst/secondary","_role":"Image"}},"url":"http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://10.204.206.10/cst/secondary","wait":0}}]
}
2015-08-13 14:56:50,708 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 3-6006676003005399182:
Processing:  { Ans: , MgmtId: 52234440052, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","downloadPct":34,"errorString":"
","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1ccfebac-b23f-3d50-9904-7722a418ece1/template/tmpl/1/4/dnld4011508956757342167tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"
","wait":0}}] }
2015-08-13 14:56:50,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d7ca7eb2)
Begin cleanup expired async-jobs
2015-08-13 14:56:50,736 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d7ca7eb2)
End cleanup expired async-jobs
2015-08-13 14:56:54,145 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f88bc1f6)
Zone 1 is ready to launch secondary storage VM
2015-08-13 14:56:54,365 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-585ce4cb)
Zone 1 is ready to launch console proxy
2015-08-13 14:57:00,170 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) Ping
from 3
2015-08-13 14:57:00,708 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-679f97b3)
Performing image store createTemplate async callback
2015-08-13 14:57:00,713 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-679f97b3)
Performing image store createTemplate async callback
2015-08-13 14:57:00,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-7d12f2d1)
Begin cleanup expired async-jobs
2015-08-13 14:57:00,735 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-7d12f2d1)
End cleanup expired async-jobs
2015-08-13 14:57:01,899 DEBUG [c.c.s.d.DownloadListener] (Timer-10:ctx-dbfe89f0) Scheduling
timeout at 30000 ms, TEMPLATE: 4 at host 3
2015-08-13 14:57:10,708 DEBUG [o.a.c.s.RemoteHostEndPoint] (Timer-10:ctx-f673d42a) Sending
command org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3
2015-08-13 14:57:10,709 DEBUG [c.c.a.t.Request] (Timer-10:ctx-f673d42a) Seq 3-6006676003005399183:
Sending  { Cmd , MgmtId: 52234440052, via: 3(s-138-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","request":"GET_STATUS","hvm":false,"description":"CentOS
5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.204.206.10/cst/secondary","_role":"Image"}},"url":"http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://10.204.206.10/cst/secondary","wait":0}}]
}
2015-08-13 14:57:10,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d8782721)
Begin cleanup expired async-jobs
2015-08-13 14:57:10,735 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d8782721)
End cleanup expired async-jobs
2015-08-13 14:57:10,754 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 3-6006676003005399183:
Processing:  { Ans: , MgmtId: 52234440052, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"a8e98b8d-5bd7-4c2b-a59b-a020379818ec","downloadPct":34,"errorString":"
","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1ccfebac-b23f-3d50-9904-7722a418ece1/template/tmpl/1/4/dnld4011508956757342167tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"
","wait":0}}] }
2015-08-13 14:57:10,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6e457831)
Found 0 routers to update status. 
2015-08-13 14:57:10,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6e457831)
Found 0 networks to update RvR status. 
2015-08-13 14:57:15,768 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Resetting hosts suitable for reconnect
2015-08-13 14:57:15,769 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Completed resetting hosts suitable for reconnect
2015-08-13 14:57:15,769 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Acquiring hosts for clusters already owned by this management server
2015-08-13 14:57:15,769 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Completed acquiring hosts for clusters already owned by this management server
2015-08-13 14:57:15,769 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Acquiring hosts for clusters not owned by any management server
2015-08-13 14:57:15,769 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-65fba20c)
Completed acquiring hosts for clusters not owned by any management server
2015-08-13 14:57:20,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c2aa49a9)
Begin cleanup expired async-jobs
2015-08-13 14:57:20,735 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c2aa49a9)
End cleanup expired async-jobs
2015-08-13 14:57:20,754 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-9:ctx-749a0094)
Performing image store createTemplate async callback
2015-08-13 14:57:20,758 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-9:ctx-749a0094)
Performing image store createTemplate async callback
2015-08-13 14:57:24,145 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f638d229)
Zone 1 is ready to launch secondary storage VM
2015-08-13 14:57:24,365 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-85eb4de4)
Zone 1 is ready to launch console proxy
2015-08-13 14:57:25,857 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-ad6bab98) AutoScaling
Monitor is running...
2015-08-13 14:57:25,872 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-f3add6f6) VmStatsCollector
is running...
^C
[root@cstmgt1 ~]# 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message