cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "edison su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-4631) [Automation] Failed to create snapshot from volume due to storage pool missing error in libvirtd
Date Mon, 09 Sep 2013 20:28:51 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762242#comment-13762242
] 

edison su commented on CLOUDSTACK-4631:
---------------------------------------

It's a similar bug related to CLOUDSTACK-2729:
Libvirt has the following code:

if (backend->refreshPool(conn, pool) < 0) {
        if (backend->stopPool)
            backend->stopPool(conn, pool);
        virStoragePoolObjRemove(&driver->pools, pool);
        pool = NULL;
        goto cleanup;
    }

in storagePoolCreate. 

During the storage creation, if refresh pool failed(the possible reason, is related to bug
2729), libvirt will stop the pool, but there are vms running on it, then can't create the
storage pool.
That's exactly happened, in QA's test environment, during the time from 07:56:55,260 to 2013-09-09
08:06:06,993. 
                
> [Automation] Failed to create snapshot from volume due to storage pool missing error
in libvirtd
> ------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4631
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4631
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: KVM, Snapshot
>    Affects Versions: 4.2.1
>         Environment: RHEL 6.3
> Automation env
> Branch : 4.2.0-forward 
>            Reporter: Rayees Namathponnan
>            Assignee: edison su
>            Priority: Critical
>             Fix For: 4.2.1
>
>         Attachments: CLOUDSTACK-4628.rar
>
>
> This issue found while running regression test case 
> integration.component.test_project_resources.TestSnapshots.test_06_create_snapshots_in_project
> Test case failing while creating snapshot, observed below error MS log; there are some
other snapshot test cases; its not effect; looks like its a inconsistent issue 
> 2013-09-09 10:45:47,445 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null)
Seq 1-1436357958: Processing:  { Ans: , MgmtId: 29066118877352, via: 1
> , Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
co
> m.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal
error Child process (/bin/umount /mnt/fff90cb5-06dd-33b3-8815-d78c08ca0
> 1d9) status unexpected: exit status 16","wait":0}}] }
> 2013-09-09 10:45:47,446 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null)
Seq 1-1436357970: Sending now.  is current sequence.
> 2013-09-09 10:45:47,446 DEBUG [agent.transport.Request] (Job-Executor-146:job-1770 =
[ 53db2ef6-6a92-4b69-bc5a-ea673814a4b9 ]) Seq 1-1436357958: Received:  {
>  Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2013-09-09 10:45:47,458 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-146:job-1770
= [ 53db2ef6-6a92-4b69-bc5a-ea673814a4b9 ]) Failed to create
> snapshot
> com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: org.libvir
> t.LibvirtException: internal error Child process (/bin/umount /mnt/fff90cb5-06dd-33b3-8815-d78c08ca01d9)
status unexpected: exit status 16
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1307)
>         at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2738)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-09-09 10:45:47,465 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-146:job-1770
= [ 53db2ef6-6a92-4b69-bc5a-ea673814a4b9 ]) Take snapshot: 498 fa
> iled
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1307)
>         at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2738)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException:
>  org.libvirt.LibvirtException: internal error Child process (/bin/umount /mnt/fff90cb5-06dd-33b3-8815-d78c08ca01d9)
status unexpected: exit status 16
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> Agent log
> 2013-09-09 10:41:59,446 DEBUG [cloud.agent.Agent] (Agent-Handler-2:null) Received response:
Seq 1-740:  { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 100010, [{"com
> .cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":1,"wait":0},"result":true,"wait":0}}]
}
> 2013-09-09 10:42:07,696 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Processing
command: com.cloud.agent.api.GetVmStatsCommand
> 2013-09-09 10:42:14,959 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
can't get storage pool
> org.libvirt.LibvirtException: Storage pool not found: no pool with matching uuid
>         at org.libvirt.ErrorHandler.processError(Unknown Source)
>         at org.libvirt.Connect.processError(Unknown Source)
>         at org.libvirt.Connect.storagePoolLookupByUUIDString(Unknown Source)
>         at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:363)
>         at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:104)
>         at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:148)
>         at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:657)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1286)
>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
>         at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-09-09 10:42:14,961 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
createStoragePool didn't find existing running pool: org.libvirt.LibvirtExcepti
> on: Storage pool not found: no pool with matching uuid, need to create it

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message