Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9BFB810D71 for ; Thu, 12 Sep 2013 09:50:54 +0000 (UTC) Received: (qmail 84503 invoked by uid 500); 12 Sep 2013 09:50:54 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 84348 invoked by uid 500); 12 Sep 2013 09:50:53 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 83994 invoked by uid 500); 12 Sep 2013 09:50:52 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 83979 invoked by uid 99); 12 Sep 2013 09:50:52 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Sep 2013 09:50:52 +0000 Date: Thu, 12 Sep 2013 09:50:52 +0000 (UTC) From: "sadhu suresh (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Closed] (CLOUDSTACK-4423) ceph:migrate instance form one primary to another primary failing MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4423?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] sadhu suresh closed CLOUDSTACK-4423. ------------------------------------ raised separate issue for the above behavior so closing the current issue https://issues.apache.org/jira/browse/CLOUDSTACK-4652 =20 > ceph:migrate instance form one primary to another primary failing=20 > ------------------------------------------------------------------ > > Key: CLOUDSTACK-4423 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-442= 3 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Management Server > Affects Versions: 4.2.0 > Reporter: sadhu suresh > Assignee: Wido den Hollander > Priority: Critical > Attachments: management-server.rar > > > 1.deploy advanced zone with kvm cluster(single cluster with ubuntu host+n= fs as primary storage) > 2.add RBD based primary storage=20 > 3.create compute offering and create a instance using this offering=20 > 4.stop the running VM which was created in step 3=20 > 5.perform migrate instance from one primary to another primary=20 > Actual result:=20 > Migrate instance is failing with below error=20 > "failed to migration: com.cloud.exception.StorageUnavailableException: Re= source [StoragePool:2] is unreachable: migrate volume failed: com.cloud.uti= ls.exception.CloudRuntimeException: Failed to copy cloudkvm/1c0f04aa-7d19-4= 2ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2 > content of management log:=20 > 2013-08-21 04:25:07,080 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina= -exec-25:null) submit async job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab= 65 ], details: AsyncJobVO {id:21, userId: 2, accountId: 2, sessionKey: null= , instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.comm= and.admin.vm.MigrateVMCmd, cmdOriginator: null, cmdInfo: {"response":"json"= ,"sessionkey":"Dle2oW4Rjxm7blpoMLnWiHE/ZVM\u003d","virtualmachineid":"69496= 3ca-4ed6-4815-af80-5a77135b97c9","cmdEventType":"VM.MIGRATE","ctxUserId":"2= ","storageid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","httpmethod":"GET","_"= :"1377019844223","ctxAccountId":"2","ctxStartEventId":"86"}, cmdVersion: 0,= callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resul= tCode: 0, result: null, initMsid: 7175246184473, completeMsid: null, lastUp= dated: null, lastPolled: null, created: null} > 2013-08-21 04:25:07,084 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exe= cutor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Executing org.= apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-21 =3D [ 921882= 56-af48-4878-8f5d-8f1d1a23ab65 ] > 2013-08-21 04:25:07,110 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state tr= ansitted from :Stopped to Migrating with event: StorageMigrationRequestedvm= 's original host id: 1 new host id: null host id before state transition: n= ull > 2013-08-21 04:25:07,193 DEBUG [storage.motion.AncientDataMotionStrategy]= (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copyA= sync inspecting src type VOLUME copyAsync inspecting dest type VOLUME > 2013-08-21 04:25:07,197 DEBUG [cache.allocator.StorageCacheRandomAllocat= or] (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Ca= n't find staging storage in zone: 1 > 2013-08-21 04:25:07,233 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424131: Sendi= ng { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011, [{"org.ap= ache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudsta= ck.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a= ","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.Primar= yDataStoreTO":{"uuid":"08d53ff3-8884-30a4-a1d1-ac621abcc688","id":3,"poolTy= pe":"RBD","host":"10.147.41.3","path":"cloudkvm","port":6789}},"name":"ROOT= -4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volume= Id":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorTy= pe":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"u= uid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore"= :{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sad= hu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"path":"= volumes/2/4","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW",= "id":4,"hypervisorType":"KVM"}},"executeInSequence":false,"wait":10800}}] } > 2013-08-21 04:25:07,571 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424131: Recei= ved: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, { CopyCmdA= nswer } } > 2013-08-21 04:25:07,571 DEBUG [storage.motion.AncientDataMotionStrategy]= (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copy = to image store failed: com.cloud.utils.exception.CloudRuntimeException: Fai= led to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-= 489d-aa56-7ed3d59e1d4e.qcow2 > 2013-08-21 04:25:07,590 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 3-891420680: Sendi= ng { Cmd , MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 100011, [{"org.ap= ache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudst= ack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22= a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":= "nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"= ROOT-4","size":8589934592,"volumeId":4,"vmName":"i-2-4-VM","accountId":2,"f= ormat":"RAW","id":4,"hypervisorType":"KVM"}},"wait":0}}] } > 2013-08-21 04:25:07,652 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 3-891420680: Recei= ved: { Ans: , MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 10, { Answer }= } > 2013-08-21 04:25:07,655 WARN [storage.datastore.ObjectInDataStoreManager= Impl] (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) = Volume 4 is not found on image store 1, so no need to delete > 2013-08-21 04:25:07,668 WARN [storage.datastore.ObjectInDataStoreManager= Impl] (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) = Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.Pr= imaryDataStoreImpl@581a95c4), no need to delete from object in store ref ta= ble > 2013-08-21 04:25:07,684 WARN [storage.datastore.ObjectInDataStoreManager= Impl] (Job-Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) = Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.Pr= imaryDataStoreImpl@53310048), no need to delete from object in store ref ta= ble > 2013-08-21 04:25:07,747 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424132: Sendi= ng { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011, [{"org.ap= ache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudst= ack.storage.to.VolumeObjectTO":{"uuid":"7aca5603-0b2c-4de6-8dee-45f192bf113= f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.Prima= ryDataStoreTO":{"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","id":2,"poolT= ype":"RBD","host":"10.147.41.3","path":"data","port":6789}},"name":"ROOT-4"= ,"size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId"= :9,"vmName":"i-2-4-VM","accountId":2,"id":9,"hypervisorType":"KVM"}},"wait"= :0}}] } > 2013-08-21 04:25:07,927 DEBUG [agent.transport.Request] (Job-Executor-9:= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424132: Recei= ved: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, { Answer }= } > 2013-08-21 04:25:07,936 INFO [storage.volume.VolumeServiceImpl] (Job-Exe= cutor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 9 is no= t referred anywhere, remove it from volumes table > 2013-08-21 04:25:07,947 ERROR [cloud.storage.VolumeManagerImpl] (Job-Exe= cutor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) migrate volume= failed:com.cloud.utils.exception.CloudRuntimeException: Failed to copy clo= udkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d= 59e1d4e.qcow2 > 2013-08-21 04:25:07,948 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Failed to m= igration: com.cloud.exception.StorageUnavailableException: Resource [Storag= ePool:2] is unreachable: migrate volume failed: com.cloud.utils.exception.C= loudRuntimeException: Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b= 7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2 > 2013-08-21 04:25:07,953 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state tr= ansitted from :Migrating to Stopped with event: AgentReportStoppedvm's orig= inal host id: 1 new host id: null host id before state transition: null > 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's act= ual total CPU: 9572 and CPU after applying overprovisioning: 9572 > 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's act= ual total RAM: 16816889856 and RAM after applying overprovisioning: 1681688= 9856 > 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release cpu= from host: 1, old used: 2000,reserved: 100, actual total: 9572, total with= overprovisioning: 9572; new used: 1900,reserved:100; movedfromreserved: fa= lse,moveToReserveredfalse > 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-= Executor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release mem= from host: 1, old used: 2013265920,reserved: 134217728, total: 16816889856= ; new used: 1879048192,reserved:134217728; movedfromreserved: false,moveToR= eserveredfalse > 2013-08-21 04:25:07,974 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Exe= cutor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unexpected exc= eption while executing org.apache.cloudstack.api.command.admin.vm.MigrateVM= Cmd > 2013-08-21 04:25:07,984 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exe= cutor-9:job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Complete async= job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ], jobStatus: 2, resultC= ode: 530, result: Error Code: 530 Error text: Failed to migration: com.clou= d.exception.StorageUnavailableException: Resource [StoragePool:2] is unreac= hable: migrate volume failed: com.cloud.utils.exception.CloudRuntimeExcepti= on: Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea= 7-d393-489d-aa56-7ed3d59e1d4e.qcow2 > 2013-08-21 04:25:10,287 DEBUG [cloud.async.AsyncJobManagerImpl] (catalin= a-exec-24:null) Async job-21 =3D [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ] c= ompleted > =20 > =20 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira