cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wido den Hollander (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-4423) ceph:migrate instance form one primary to another primary failing
Date Thu, 22 Aug 2013 06:10:51 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wido den Hollander updated CLOUDSTACK-4423:
-------------------------------------------

    Assignee: Wido den Hollander
    
> ceph:migrate instance form one primary to another primary failing 
> ------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4423
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4423
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.2.0
>            Reporter: sadhu suresh
>            Assignee: Wido den Hollander
>            Priority: Critical
>         Attachments: management-server.rar
>
>
> 1.deploy advanced zone with kvm cluster(single cluster with ubuntu host+nfs as primary
storage)
>  2.add RBD based primary storage 
> 3.create compute offering and create a instance using this offering 
> 4.stop the running VM which was created in step 3 
> 5.perform migrate instance from one primary to another primary 
> Actual result: 
> Migrate instance is failing with below error 
> "failed to migration: com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2]
is unreachable: migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: Failed
to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> content of management log: 
> 2013-08-21 04:25:07,080 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-25:null)
submit async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ], details: AsyncJobVO {id:21,
userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd,
cmdOriginator: null, cmdInfo: {"response":"json","sessionkey":"Dle2oW4Rjxm7blpoMLnWiHE/ZVM\u003d","virtualmachineid":"694963ca-4ed6-4815-af80-5a77135b97c9","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","httpmethod":"GET","_":"1377019844223","ctxAccountId":"2","ctxStartEventId":"86"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 7175246184473, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
>  2013-08-21 04:25:07,084 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
for job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]
>  2013-08-21 04:25:07,110 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state transitted from :Stopped to Migrating
with event: StorageMigrationRequestedvm's original host id: 1 new host id: null host id before
state transition: null
>  2013-08-21 04:25:07,193 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copyAsync inspecting src type VOLUME copyAsync
inspecting dest type VOLUME
>  2013-08-21 04:25:07,197 DEBUG [cache.allocator.StorageCacheRandomAllocator] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Can't find staging storage in zone: 1
>  2013-08-21 04:25:07,233 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 1-462424131: Sending { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"08d53ff3-8884-30a4-a1d1-ac621abcc688","id":3,"poolType":"RBD","host":"10.147.41.3","path":"cloudkvm","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"path":"volumes/2/4","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"executeInSequence":false,"wait":10800}}]
}
>  2013-08-21 04:25:07,571 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 1-462424131: Received: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10,
{ CopyCmdAnswer } }
>  2013-08-21 04:25:07,571 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copy to image store failed: com.cloud.utils.exception.CloudRuntimeException:
Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
>  2013-08-21 04:25:07,590 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 3-891420680: Sending { Cmd , MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"wait":0}}]
}
>  2013-08-21 04:25:07,652 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 3-891420680: Received: { Ans: , MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 10,
{ Answer } }
>  2013-08-21 04:25:07,655 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 4 is not found on image store 1, so no
need to delete
>  2013-08-21 04:25:07,668 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@581a95c4),
no need to delete from object in store ref table
>  2013-08-21 04:25:07,684 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@53310048),
no need to delete from object in store ref table
>  2013-08-21 04:25:07,747 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 1-462424132: Sending { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7aca5603-0b2c-4de6-8dee-45f192bf113f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","id":2,"poolType":"RBD","host":"10.147.41.3","path":"data","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":9,"vmName":"i-2-4-VM","accountId":2,"id":9,"hypervisorType":"KVM"}},"wait":0}}]
}
>  2013-08-21 04:25:07,927 DEBUG [agent.transport.Request] (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
]) Seq 1-462424132: Received: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10,
{ Answer } }
>  2013-08-21 04:25:07,936 INFO [storage.volume.VolumeServiceImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 9 is not referred anywhere, remove it from
volumes table
>  2013-08-21 04:25:07,947 ERROR [cloud.storage.VolumeManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) migrate volume failed:com.cloud.utils.exception.CloudRuntimeException:
Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
>  2013-08-21 04:25:07,948 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Failed to migration: com.cloud.exception.StorageUnavailableException:
Resource [StoragePool:2] is unreachable: migrate volume failed: com.cloud.utils.exception.CloudRuntimeException:
Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
>  2013-08-21 04:25:07,953 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state transitted from :Migrating to Stopped
with event: AgentReportStoppedvm's original host id: 1 new host id: null host id before state
transition: null
>  2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's actual total CPU: 9572 and CPU after applying
overprovisioning: 9572
>  2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's actual total RAM: 16816889856 and RAM
after applying overprovisioning: 16816889856
>  2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release cpu from host: 1, old used: 2000,reserved:
100, actual total: 9572, total with overprovisioning: 9572; new used: 1900,reserved:100; movedfromreserved:
false,moveToReserveredfalse
>  2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release mem from host: 1, old used: 2013265920,reserved:
134217728, total: 16816889856; new used: 1879048192,reserved:134217728; movedfromreserved:
false,moveToReserveredfalse
>  2013-08-21 04:25:07,974 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
>  2013-08-21 04:25:07,984 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Complete async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to migration:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable:
migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: Failed to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5
to 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
>  2013-08-21 04:25:10,287 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null)
Async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ] completed
>  
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message