Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7B14910EF2 for ; Thu, 31 Oct 2013 04:54:28 +0000 (UTC) Received: (qmail 48592 invoked by uid 500); 31 Oct 2013 04:54:27 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 48554 invoked by uid 500); 31 Oct 2013 04:54:26 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 48535 invoked by uid 500); 31 Oct 2013 04:54:26 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 48528 invoked by uid 99); 31 Oct 2013 04:54:26 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Oct 2013 04:54:26 +0000 Date: Thu, 31 Oct 2013 04:54:25 +0000 (UTC) From: "Abhinandan Prateek (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-5008) [VMWARE]Failed to start the VM after performing Cold Migration of Volume to Second Zone wide primary Storage MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-5008?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinandan Prateek updated CLOUDSTACK-5008: ------------------------------------------- Assignee: Sateesh Chodapuneedi > [VMWARE]Failed to start the VM after performing Cold Migration of Volume = to Second Zone wide primary Storage > -------------------------------------------------------------------------= ----------------------------------- > > Key: CLOUDSTACK-5008 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-500= 8 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Storage Controller, VMware > Affects Versions: 4.2.1 > Reporter: Sailaja Mada > Assignee: Sateesh Chodapuneedi > Priority: Critical > Fix For: 4.2.1 > > Attachments: migrationlogs.rar > > > Steps: > 1. Configure Adv zone with VMWARE ESXi 5.0 Update 2 hypervisor=20 > 2. Configure two Zone wide primary storages=20 > 3. Have 2 VMWARE clusters each with 5.0 Update2 ESXi hosts > 4. Deploy VM using user account=20 > 5. Attach 3 DATA volumes. Write DATA onto first DISK=20 > 6. Stop the VM=20 > 7. Migrate the DATA DISK 1 to second zone wide primary storage > 8. Tried to start the VM after migration is completed.=20 > Observation:=20 > [VMWARE]Failed to start the VM after performing Cold Migration of Volume = to Second Zone wide primary Storage > 2013-10-30 19:49:49,187 WARN [vmware.resource.VmwareResource] (DirectAge= nt-267:10.102.192.19) StartCommand failed due to Exception: java.lang.Runti= meException > Message: File []/vmfs/volumes/371681b9-ed4b0743/i-4-10-VM/9db1d292e0394eb= 39e69c8adee09e26c.vmdk was not found > java.lang.RuntimeException: File []/vmfs/volumes/371681b9-ed4b0743/i-4-10= -VM/9db1d292e0394eb39e69c8adee09e26c.vmdk was not found > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(Vmwa= reClient.java:411) > at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.configureVm(Vi= rtualMachineMO.java:843) > at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(Vm= wareResource.java:2966) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeReq= uest(VmwareResource.java:513) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgen= tAttache.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-10-30 19:49:49,194 DEBUG [agent.manager.DirectAgentAttache] (DirectA= gent-267:null) Seq 8-1057948407: Response Received: > 2013-10-30 19:49:49,198 DEBUG [agent.transport.Request] (DirectAgent-267:= null) Seq 8-1057948407: Processing: { Ans: , MgmtId: 94838926819810, via: = 8, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":10,= "name":"i-4-10-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":500= ,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"hostName":"sharedins= t1","arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash= ":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":fal= se,"vncPassword":"4f62f2fc9be065a7","params":{"nicAdapter":"E1000","vmware.= reserve.cpu":"false","nestedVirtualizationFlag":"false","Message.ReservedCa= pacityFreed.Flag":"false","rootDiskController":"ide","vmware.reserve.mem":"= false"},"uuid":"faa8546b-85e5-4fa1-8a7a-c0fbb476c649","disks":[{"data":{"or= g.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a6d0d1ac-2791-40f0-= a9f5-26b524a45972","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.= storage.to.PrimaryDataStoreTO":{"uuid":"29cbd2fe-d168-3f80-b380-08ca5c399ba= 7","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/c= pg_vol/sailaja/newps1","port":2049}},"name":"ROOT-10","size":2147483648,"pa= th":"ROOT-10","volumeId":10,"vmName":"i-4-10-VM","accountId":4,"chainInfo":= "{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[29cbd2fed1683f80b38008= ca5c399ba7] i-4-10-VM/ROOT-10.vmdk\"]}","format":"OVA","id":10,"hypervisorT= ype":"VMware"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.= storage.to.VolumeObjectTO":{"uuid":"0b57572c-5f4e-42f9-9d9d-01b1e96698ea","= volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.Prima= ryDataStoreTO":{"uuid":"29cbd2fe-d168-3f80-b380-08ca5c399ba7","id":1,"poolT= ype":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/n= ewps1","port":2049}},"name":"newd1","size":5368709120,"path":"0a46a41a02e04= 5d0a310d4b2d9e56b9f","volumeId":18,"vmName":"i-4-10-VM","accountId":4,"chai= nInfo":"{\"diskDeviceBusName\":\"scsi0:1\",\"diskChain\":[\"[29cbd2fed1683f= 80b38008ca5c399ba7] i-4-10-VM/0a46a41a02e045d0a310d4b2d9e56b9f.vmdk\"]}","f= ormat":"OVA","id":18,"hypervisorType":"VMware"}},"diskSeq":2,"type":"DATADI= SK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b5= 8f7371-2570-49c2-b9f3-0ef412e2a260","volumeType":"DATADISK","dataStore":{"o= rg.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"29cbd2fe-d168-= 3f80-b380-08ca5c399ba7","id":1,"poolType":"NetworkFilesystem","host":"10.10= 2.192.100","path":"/cpg_vol/sailaja/newps1","port":2049}},"name":"newd2","s= ize":5368709120,"path":"eee0f43fdfbc4572821c5f5a9fc32678","volumeId":19,"vm= Name":"i-4-10-VM","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"scsi0= :2\",\"diskChain\":[\"[29cbd2fed1683f80b38008ca5c399ba7] i-4-10-VM/eee0f43f= dfbc4572821c5f5a9fc32678.vmdk\"]}","format":"OVA","id":19,"hypervisorType":= "VMware"}},"diskSeq":4,"type":"DATADISK"},{"data":{"org.apache.cloudstack.s= torage.to.VolumeObjectTO":{"uuid":"313118e2-a6a0-43ad-acbb-a5fe0ac3de1d","v= olumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.Primar= yDataStoreTO":{"uuid":"25505c51-4b14-3f06-b3fe-9c36fe69a753","id":2,"poolTy= pe":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/cl= uster2ps1","port":2049}},"name":"DATA-10","size":5368709120,"path":"75a1abc= cb8684639a12f615116e68b24","volumeId":20,"vmName":"i-4-10-VM","accountId":4= ,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[29cbd2fe= d1683f80b38008ca5c399ba7] i-4-10-VM/9db1d292e0394eb39e69c8adee09e26c.vmdk\"= ]}","format":"OVA","id":20,"hypervisorType":"VMware"}},"diskSeq":1,"type":"= DATADISK"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"i= d":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],= "nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"a5507= bf6-5ccd-4d33-b21e-41acbfd7b027","ip":"10.102.198.12","netmask":"255.255.25= 5.128","gateway":"10.102.198.1","mac":"06:cd:f4:00:00:23","dns1":"10.103.12= 8.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://110","is= olationUri":"vlan://110","isSecurityGroupEnabled":false}]},"result":false,"= details":"StartCommand failed due to Exception: java.lang.RuntimeException\= nMessage: File []/vmfs/volumes/371681b9-ed4b0743/i-4-10-VM/9db1d292e0394eb3= 9e69c8adee09e26c.vmdk was not found\n","wait":0}}] } > 2013-10-30 19:49:49,198 DEBUG [agent.transport.Request] (Job-Executor-126= :job-103 =3D [ 3c563876-b27a-4bf3-bc90-dbf2aaa98628 ]) Seq 8-1057948407: Re= ceived: { Ans: , MgmtId: 94838926819810, via: 8, Ver: v1, Flags: 110, { St= artAnswer } } > 2013-10-30 19:49:49,200 DEBUG [agent.manager.AgentAttache] (DirectAgent-2= 67:null) Seq 8-1057948407: No more commands found > 2013-10-30 19:49:49,204 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-E= xecutor-126:job-103 =3D [ 3c563876-b27a-4bf3-bc90-dbf2aaa98628 ]) Unable to= start VM on Host[-8-Routing] due to StartCommand failed due to Exception: = java.lang.RuntimeException > Message: File []/vmfs/volumes/371681b9-ed4b0743/i-4-10-VM/9db1d292e0394eb= 39e69c8adee09e26c.vmdk was not found > 2013-10-30 19:49:49,214 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-E= xecutor-126:job-103 =3D [ 3c563876-b27a-4bf3-bc90-dbf2aaa98628 ]) Cleaning = up resources for the vm VM[User|sharedinst1] in Starting state > 2013-10-30 19:49:49,216 DEBUG [agent.transport.Request] (Job-Executor-126= :job-103 =3D [ 3c563876-b27a-4bf3-bc90-dbf2aaa98628 ]) Seq 8-1057948408: Se= nding { Cmd , MgmtId: 94838926819810, via: 8, Ver: v1, Flags: 100111, [{"c= om.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"= vmName":"i-4-10-VM","wait":0}}] } > 2013-10-30 19:49:49,216 DEBUG [agent.transport.Request] (Job-Executor-126= :job-103 =3D [ 3c563876-b27a-4bf3-bc90-dbf2aaa98628 ]) Seq 8-1057948408: Ex= ecuting: { Cmd , MgmtId: 94838926819810, via: 8, Ver: v1, Flags: 100111, [= {"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":tru= e,"vmName":"i-4-10-VM","wait":0}}] } > 2013-10-30 19:49:49,216 DEBUG [agent.manager.DirectAgentAttache] (DirectA= gent-197:null) Seq 8-1057948408: Executing request > 2013-10-30 19:49:49,217 INFO [vmware.resource.VmwareResource] (DirectAge= nt-197:10.102.192.19) Executing resource StopCommand: {"isProxy":false,"exe= cuteInSequence":true,"vmName":"i-4-10-VM","wait":0} > 2013-10-30 19:49:50,841 DEBUG [agent.manager.DirectAgentAttache] (DirectA= gent-366:null) Ping from 7 > 2013-10-30 19:49:50,980 WARN [vmware.resource.VmwareResource] (DirectAge= nt-122:10.102.192.20) StartCommand failed due to Exception: java.lang.Runti= meException > Message: The name 'i-4-10-VM' already exists. > java.lang.RuntimeException: The name 'i-4-10-VM' already exists. > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(Vmwa= reClient.java:411) > at com.cloud.hypervisor.vmware.mo.HostMO.createVm(HostMO.java:574= ) > at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.createBlan= kVm(HypervisorHostHelper.java:1184) > at com.cloud.hypervisor.vmware.mo.HostMO.createBlankVm(HostMO.jav= a:761) > at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(Vm= wareResource.java:2717) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeReq= uest(VmwareResource.java:513) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgen= tAttache.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-10-30 19:37:43,310 INFO [cloud.server.ManagementServerImpl] (catali= na-exec-16:null) Volume Vol[11|vm=3D10|DATADISK] isn't attached to any runn= ing vm. Looking for storage pools in the cluster to which this volumes can = be migrated. > 2013-10-30 19:37:45,696 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:nu= ll) =3D=3D=3DSTART=3D=3D=3D 10.104.255.45 -- GET command=3DmigrateVolume&= livemigrate=3Dtrue&storageid=3D25505c51-4b14-3f06-b3fe-9c36fe69a753&volumei= d=3D5f4fef52-c7e3-4a44-bb86-40005f2fa763&response=3Djson&sessionkey=3DaBW1O= 134QVprgEZEHjdVdGfH6cY%3D&_=3D1383142396380 > 2013-10-30 19:37:45,770 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina= -exec-21:null) submit async job-102 =3D [ 0af73b05-444e-4a19-a82a-045852e8e= 61b ], details: AsyncJobVO {id:102, userId: 2, accountId: 2, sessionKey: nu= ll, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.co= mmand.user.volume.MigrateVolumeCmd, cmdOriginator: null, cmdInfo: {"respons= e":"json","sessionkey":"aBW1O134QVprgEZEHjdVdGfH6cY\u003d","cmdEventType":"= VOLUME.MIGRATE","ctxUserId":"2","storageid":"25505c51-4b14-3f06-b3fe-9c36fe= 69a753","livemigrate":"true","httpmethod":"GET","volumeid":"5f4fef52-c7e3-4= a44-bb86-40005f2fa763","_":"1383142396380","ctxAccountId":"2","ctxStartEven= tId":"403"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status:= 0, processStatus: 0, resultCode: 0, result: null, initMsid: 94838926819810= , completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-10-30 19:37:45,772 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:nu= ll) =3D=3D=3DEND=3D=3D=3D 10.104.255.45 -- GET command=3DmigrateVolume&li= vemigrate=3Dtrue&storageid=3D25505c51-4b14-3f06-b3fe-9c36fe69a753&volumeid= =3D5f4fef52-c7e3-4a44-bb86-40005f2fa763&response=3Djson&sessionkey=3DaBW1O1= 34QVprgEZEHjdVdGfH6cY%3D&_=3D1383142396380 > mysql> select * from volumes where uuid=3D'5f4fef52-c7e3-4a44-bb86-40005f= 2fa763'\G; > *************************** 1. row *************************** > id: 11 > account_id: 4 > domain_id: 1 > pool_id: 1 > last_pool_id: NULL > instance_id: 10 > device_id: 1 > name: DATA-10 > uuid: 5f4fef52-c7e3-4a44-bb86-40005f2fa763 > size: 5368709120 > folder: NULL > path: 9db1d292e0394eb39e69c8adee09e26c > pod_id: NULL > data_center_id: 2 > iscsi_name: NULL > host_ip: NULL > volume_type: DATADISK > pool_type: NULL > disk_offering_id: 3 > template_id: NULL > first_snapshot_backup_uuid: NULL > recreatable: 0 > created: 2013-10-30 05:52:24 > attached: NULL > updated: 2013-10-30 14:16:29 > removed: 2013-10-30 14:16:29 > state: Expunged > chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["= [29cbd2fed1683f80b38008ca5c399ba7] i-4-10-VM/9db1d292e0394eb39e69c8adee09e2= 6c.vmdk"]} > update_count: 8 > disk_type: NULL > vm_snapshot_chain_size: NULL > iso_id: NULL > display_volume: 0 > format: OVA > min_iops: NULL > max_iops: NULL > 1 row in set (0.00 sec) > ERROR: > No query specified > mysql> -- This message was sent by Atlassian JIRA (v6.1#6144)