Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DCD051057D for ; Tue, 3 Sep 2013 06:10:53 +0000 (UTC) Received: (qmail 35520 invoked by uid 500); 3 Sep 2013 06:10:53 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 35124 invoked by uid 500); 3 Sep 2013 06:10:53 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 35029 invoked by uid 500); 3 Sep 2013 06:10:52 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 35023 invoked by uid 99); 3 Sep 2013 06:10:52 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 06:10:52 +0000 Date: Tue, 3 Sep 2013 06:10:52 +0000 (UTC) From: "Sateesh Chodapuneedi (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-4587) VM is failing to deploy on a Legacy zone after adding zone wide primary storage and moving cluster wide primary storage to maintenance mode MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sateesh Chodapuneedi updated CLOUDSTACK-4587: --------------------------------------------- Assignee: (was: Sateesh Chodapuneedi) > VM is failing to deploy on a Legacy zone after adding zone wide primary storage and moving cluster wide primary storage to maintenance mode > -------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-4587 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4587 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Storage Controller > Affects Versions: 4.2.1 > Reporter: Sailaja Mada > Priority: Critical > Fix For: 4.2.1 > > Attachments: vmdeploylogs.rar > > > Steps: > 1. Upgraded from 307 to 4.2 using VMWARE Cluster with Cluster level primary storage > 2. Add Zone wide primary storage after upgrade > 3. Move the cluster level scope storage to Maintenance mode > 4. Try to deploy new VM. > Observation: > VM is failing to deploy on a Legacy zone after adding zone wide primary storage and moving cluster wide primary storage to maintenance mode > 2013-09-01 14:21:10,323 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Network id=207 is already implemented > 2013-09-01 14:21:10,354 DEBUG [network.guru.PodBasedNetworkGuru] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Allocated a nic NicProfile[119-35-4a8f547b-7bff-4b3f-ba10-b0146af4d1bb-10.102.195.111-null for VM[DomainRouter|r-35-VM] > 2013-09-01 14:21:10,390 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Checking if we need to prepare 1 volumes for VM[DomainRouter|r-35-VM] > 2013-09-01 14:21:10,400 DEBUG [storage.image.TemplateDataFactoryImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) template 203 is already in store:2, type:Image > 2013-09-01 14:21:10,409 DEBUG [storage.image.TemplateDataFactoryImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) template 203 is already in store:203, type:Primary > 2013-09-01 14:21:10,410 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Found template 203-2-3cfe22ca-3a33-39a0-bf5e-0dab154869fd in storage pool 203 with VMTemplateStoragePool id: 11 > 2013-09-01 14:21:10,421 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Acquire lock on VMTemplateStoragePool 11 with timeout 3600 seconds > 2013-09-01 14:21:10,423 INFO [storage.volume.VolumeServiceImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) lock is acquired for VMTemplateStoragePool 11 > 2013-09-01 14:21:10,431 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE > 2013-09-01 14:21:10,519 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) copy object failed: > java.lang.NullPointerException > at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210) > at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411) > at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:440) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:569) > at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2536) > at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950) > at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188) > at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034) > at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-01 14:21:10,520 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) copy failed > com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException > at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231) > at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411) > at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:440) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:569) > at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2536) > at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950) > at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188) > at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034) > at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-01 14:21:10,525 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-152:null) Seq 5-1348865999: Response Received: > 2013-09-01 14:21:10,525 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 5-1348865999: Received: { Ans: , MgmtId: 227594284004867, via: 5, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > 2013-09-01 14:21:10,530 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-172:null) Seq 6-1919947690: Executing request > 2013-09-01 14:21:10,558 INFO [storage.volume.VolumeServiceImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) releasing lock for VMTemplateStoragePool 11 > 2013-09-01 14:21:10,558 WARN [utils.db.Merovingian2] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Was unable to find lock for the key template_spool_ref11 and thread id 595423976 > 2013-09-01 14:21:10,558 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable to create Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException > 2013-09-01 14:21:10,558 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable to contact resource. > 2013-09-01 14:21:10,558 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable to create Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException > 2013-09-01 14:21:10,558 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable to contact resource. > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:203] is unreachable: Unable to create Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException > at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544) > at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950) > at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188) > at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034) > at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-01 14:21:10,592 DEBUG [vmware.resource.VmwareResource] (DirectAgent-172:10.102.192.20) Datastore summary info, storageId: 55e6d095-7e22-3f4a-b533-874a9fede8df, localPath: /cpg_vol/sailaja/vmwareps1, poolType: NetworkFilesystem, capacity: 879609303040, free: 376848912384, used: 502760390656 > 2013-09-01 14:21:10,679 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-172:null) Seq 6-1919947690: Response Received: > 2013-09-01 14:21:10,679 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 6-1919947690: Received: { Ans: , MgmtId: 227594284004867, via: 6, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > 2013-09-01 14:21:10,685 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-271:null) Seq 6-1919947691: Executing request > 2013-09-01 14:21:10,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Cleaning up resources for the vm VM[DomainRouter|r-35-VM] in Starting state > 2013-09-01 14:21:10,696 DEBUG [agent.transport.Request] (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Seq 6-1919947692: Sending { Cmd , MgmtId: 227594284004867, via: 6, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"r-35-VM","wait":0}}] } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira