Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8129E1046A for ; Sun, 15 Sep 2013 08:51:01 +0000 (UTC) Received: (qmail 17566 invoked by uid 500); 15 Sep 2013 08:50:57 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 17555 invoked by uid 500); 15 Sep 2013 08:50:57 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 17536 invoked by uid 500); 15 Sep 2013 08:50:55 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 17525 invoked by uid 99); 15 Sep 2013 08:50:55 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Sep 2013 08:50:55 +0000 Date: Sun, 15 Sep 2013 08:50:55 +0000 (UTC) From: "Sailaja Mada (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CLOUDSTACK-4663) [VMWARE][ZWPS]Failed to start one of the VM if two stopped VM's are started in parallel MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailaja Mada resolved CLOUDSTACK-4663. -------------------------------------- Resolution: Fixed Assignee: Prachi Damle This issue got fixed with below commit : Commit f3be68a83301837d14243fec8eba9927565b928e in branch refs/heads/4.2-forward from Prachi Damle [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=f3be68a ] Hence closing the ticket. > [VMWARE][ZWPS]Failed to start one of the VM if two stopped VM's are started in parallel > --------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-4663 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4663 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Storage Controller, VMware > Affects Versions: 4.2.1 > Reporter: Sailaja Mada > Assignee: Prachi Damle > Priority: Critical > Fix For: 4.2.1 > > Attachments: alllogs.rar > > > Steps: > 1. Configure VMWARE with 2 zone wide primary storages > 2. Deploy 2 VM's > 3. Add Data Volume to instance 2 , Migrate ROOT/DATA volume of this instance to second Zone wide primary storage > 3. Stop these 2 VM's > 4. Try to start these 2 VM's 1 after the other > Observation: > [VMWARE][ZWPS]Failed to start one of the VM if two stopped VM's are started in parallel > 2013-09-13 14:04:13,039 INFO [storage.resource.VmwareStorageLayoutHelper] (DirectAgent-109:10.102.192.18) Check if we need to move [40cdbd8d08d1310aa1e469e5d542bca1] i-3-6-VM/d327efa6b8e7480e99cbdfff2c2b9810.vmdk to its root location > 2013-09-13 14:04:13,045 INFO [storage.resource.VmwareStorageLayoutHelper] (DirectAgent-109:10.102.192.18) Move [40cdbd8d08d1310aa1e469e5d542bca1] i-3-6-VM/d327efa6b8e7480e99cbdfff2c2b9810.vmdk -> [40cdbd8d08d1310aa1e469e5d542bca1] d327efa6b8e7480e99cbdfff2c2b9810.vmdk > 2013-09-13 14:04:13,245 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===START=== 10.144.6.10 -- GET command=queryAsyncJobResult&jobId=70bee8e8-080d-4db8-82b4-7f64fafb490d&response=json&sessionkey=CiTQg9ZZ2%2FebPatcQTFgfdAJYDM%3D&_=1379061530062 > 2013-09-13 14:04:13,250 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-109:10.102.192.18) delete volume failed due to Exception: java.lang.RuntimeException > Message: File [40cdbd8d08d1310aa1e469e5d542bca1] i-3-6-VM/d327efa6b8e7480e99cbdfff2c2b9810.vmdk was not found > java.lang.RuntimeException: File [40cdbd8d08d1310aa1e469e5d542bca1] i-3-6-VM/d327efa6b8e7480e99cbdfff2c2b9810.vmdk was not found > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:378) > at com.cloud.hypervisor.vmware.mo.DatastoreMO.moveDatastoreFile(DatastoreMO.java:235) > at com.cloud.storage.resource.VmwareStorageLayoutHelper.moveVolumeToRootFolder(VmwareStorageLayoutHelper.java:183) > at com.cloud.storage.resource.VmwareStorageProcessor.deleteVolume(VmwareStorageProcessor.java:1512) > at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:114) > at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:53) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:565) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-13 14:04:13,257 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-109:null) Seq 1-872223588: Response Received: > 2013-09-13 14:04:13,258 DEBUG [agent.transport.Request] (DirectAgent-109:null) Seq 1-872223588: Processing: { Ans: , MgmtId: 94838926819810, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"delete volume failed due to Exception: java.lang.RuntimeException\nMessage: File [40cdbd8d08d1310aa1e469e5d542bca1] i-3-6-VM/d327efa6b8e7480e99cbdfff2c2b9810.vmdk was not found\n","wait":0}}] } > 2013-09-13 14:04:13,258 DEBUG [agent.transport.Request] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Seq 1-872223588: Received: { Ans: , MgmtId: 94838926819810, via: 1, Ver: v1, Flags: 10, { Answer } } > 2013-09-13 14:04:13,269 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7f2c2908), no need to delete from object in store ref table > 2013-09-13 14:04:13,270 ERROR [cloud.storage.VolumeManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) migrate volume failed:copy volume from primary to secondary failed due to exception: Exception: java.lang.RuntimeException > Message: File [bdeb0a5deabb386c8a08bffa7138b277] i-3-6-VM/i-3-6-VM.vmx was not found > 2013-09-13 14:04:13,270 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Unable to contact resource. > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: migrate volume failed: copy volume from primary to secondary failed due to exception: Exception: java.lang.RuntimeException > Message: File [bdeb0a5deabb386c8a08bffa7138b277] i-3-6-VM/i-3-6-VM.vmx was not found > at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2263) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2609) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:889) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:1948) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-13 14:04:13,284 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Cleaning up resources for the vm VM[User|instance2] in Starting state > 2013-09-13 14:04:13,289 DEBUG [agent.transport.Request] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Seq 1-872223590: Waiting for Seq 872223589 Scheduling: { Cmd , MgmtId: 94838926819810, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-3-6-VM","wait":0}}] } > 2013-09-13 14:04:13,294 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END=== > 2013-09-13 14:04:34,310 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1 > 2013-09-13 14:04:34,314 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Removing from the clusterId list these clusters from avoid set: [1] > 2013-09-13 14:04:34,315 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) No clusters found after removing disabled clusters and clusters in avoid list, returning. > 2013-09-13 14:04:34,349 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: null > 2013-09-13 14:04:34,377 INFO [user.vm.StartVMCmd] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|instance2]Scope=interface com.cloud.dc.DataCenter; id=1 > 2013-09-13 14:04:34,377 INFO [user.vm.StartVMCmd] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Unable to create a deployment for VM[User|instance2] > com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|instance2]Scope=interface com.cloud.dc.DataCenter; id=1 > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:1948) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-09-13 14:04:34,380 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-39:job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ]) Complete async job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ], jobStatus: 2, resultCode: 530, result: Error Code: 533 Error text: Unable to create a deployment for VM[User|instance2] > 2013-09-13 14:04:34,454 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===START=== 10.144.6.10 -- GET command=queryAsyncJobResult&jobId=70bee8e8-080d-4db8-82b4-7f64fafb490d&response=json&sessionkey=CiTQg9ZZ2%2FebPatcQTFgfdAJYDM%3D&_=1379061551304 > 2013-09-13 14:04:34,466 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-4:null) Async job-38 = [ 70bee8e8-080d-4db8-82b4-7f64fafb490d ] completed > 2013-09-13 14:04:34,472 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===END=== 10.144.6.10 -- GET command=queryAsyncJobResult&jobId=70bee8e8-080d-4db8-82b4-7f64fafb490d&response=json&sessionkey=CiTQg9ZZ2%2FebPatcQTFgfdAJYDM%3D&_=1379061551304 > 2013-09-13 14:04:37,277 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===START=== 10.144.6.10 -- GET command=queryAsyncJobResult&jobId=0c1c2d57-b817-4c46-abe5-09a25eb423ea&response=json&sessionkey=CiTQg9ZZ2%2FebPatcQTFgfdAJYDM%3D&_=1379061554127 > 2013-09-13 14:04:37,289 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-2:null) Async job-36 = [ 0c1c2d57-b817-4c46-abe5-09a25eb423ea ] completed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira