cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjeev N (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-4211) [Object_store_refactor] Failed to create snapshot with S3 on kvm
Date Fri, 09 Aug 2013 12:07:48 GMT
Sanjeev N created CLOUDSTACK-4211:
-------------------------------------

             Summary: [Object_store_refactor] Failed to create snapshot with S3 on kvm
                 Key: CLOUDSTACK-4211
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4211
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: KVM, Snapshot, Storage Controller
    Affects Versions: 4.2.0
         Environment: Latest build from ACS 4.2 branch
Storage: S3 for secondary, NFS for secondary staging, and local storage for primary
Cluster: KVM
            Reporter: Sanjeev N
            Priority: Critical
             Fix For: 4.2.0


 Failed to create snapshot with S3 on kvm

Steps to Reproduce:
=================
1. Bring up CS with KVM cluster using  S3 for secondary, NFS for secondary staging, and local
storage for primary
2.Deploy guest vm using default cent os template with both root and data disk
3.Create snapshot on root/data disk

Result:
=====
Snapshot creation failed with CloudRuntimeException: unsupported protocol

Observations:
============
Snapshot is created on primary but failed to copy to secondary staging storage.

Following is the log snippet:
2013-08-09 13:28:25,757 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null) ===START===  10.146.0.20
-- GET  command=createSnapshot&volumeid=2b3fdf2f-579b-4771-8ac8-3d7ef2988a79&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049616400
2013-08-09 13:28:25,956 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-6:null) submit
async job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ], details: AsyncJobVO {id:17, userId:
3, accountId: 3, sessionKey: null, instanceType: Snapshot, instanceId: 3, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo: {"id":"3","response":"json","sessionkey":"zXXHX4Nf5yJtILkB4BA8E75f9HA\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"3","httpmethod":"GET","_":"1376049616400","volumeid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","ctxAccountId":"3","ctxStartEventId":"79"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2013-08-09 13:28:25,959 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null) ===END===  10.146.0.20
-- GET  command=createSnapshot&volumeid=2b3fdf2f-579b-4771-8ac8-3d7ef2988a79&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049616400
2013-08-09 13:28:25,962 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-2:job-17 = [
0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd
for job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]
2013-08-09 13:28:26,039 INFO  [user.snapshot.CreateSnapshotCmd] (Job-Executor-2:job-17 = [
0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) VOLSS: createSnapshotCmd starts:1376069306039
2013-08-09 13:28:26,313 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 1-1681129511: Sending  { Cmd , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"wait":0}}]
}
2013-08-09 13:28:27,219 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq
1-1681129511: Processing:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf","id":0}},"result":true,"wait":0}}]
}
2013-08-09 13:28:27,219 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 1-1681129511: Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10,
{ CreateObjectAnswer } }
2013-08-09 13:28:27,284 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) copyAsync inspecting src type SNAPSHOT copyAsync
inspecting dest type SNAPSHOT
2013-08-09 13:28:27,330 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) needCacheStorage true, dest at snapshots/3/7 dest
role Image/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf
src role Primary
2013-08-09 13:28:27,433 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 1-1681129512: Sending  { Cmd , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"28c97b44-c81a-41db-9644-1f28c674d121","endPoint":"10.147.29.56:8080","bucketName":"imagestore119","httpsFlag":false,"created":"Aug
9, 2013 12:09:22 PM","enableRRS":false}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os","_role":"ImageCache"}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"executeInSequence":false,"wait":21600}}]
}
2013-08-09 13:28:27,444 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq
1-1681129512: Processing:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"unsupported
protocol","wait":0}}] }
2013-08-09 13:28:27,444 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 1-1681129512: Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10,
{ CopyCmdAnswer } }
2013-08-09 13:28:27,462 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 3-967704583: Sending  { Cmd , MgmtId: 7332683579487, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os","_role":"ImageCache"}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"wait":0}}]
}
2013-08-09 13:28:27,784 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-12:null)
SeqA 2-354: Processing Seq 2-354:  { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
 \"connections\": [\n    {\n      \"id\": 2,\n      \"clientInfo\": \"\",\n      \"host\":
\"10.147.40.8\",\n      \"port\": 5905,\n      \"tag\": \"5084fb7b-0113-4574-85e5-f85618a97a04\",\n
     \"createTime\": 1376049502150,\n      \"lastUsedTime\": 1376049608916\n    }\n  ]\n}","wait":0}}]
}
2013-08-09 13:28:27,798 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-12:null)
SeqA 2-354: Sending Seq 2-354:  { Ans: , MgmtId: 7332683579487, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-09 13:28:27,905 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null) Seq
3-967704583: Processing:  { Ans: , MgmtId: 7332683579487, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot
directory 3 doesn't exist","wait":0}}] }
2013-08-09 13:28:27,906 DEBUG [agent.transport.Request] (Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
]) Seq 3-967704583: Received:  { Ans: , MgmtId: 7332683579487, via: 3, Ver: v1, Flags: 10,
{ Answer } }
2013-08-09 13:28:27,940 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: unsupported protocol
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:286)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:137)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:256)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1004)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1256)
        at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2674)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
2013-08-09 13:28:27,954 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-2:job-17 =
[ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Take snapshot: 7 failed: com.cloud.utils.exception.CloudRuntimeException:
Failed to create snapshot
2013-08-09 13:28:27,955 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) VmStatsCollector
is running...
2013-08-09 13:28:27,984 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-2:job-17 = [
0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Complete async job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1
], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot
due to an internal error creating snapshot for volume 7
2013-08-09 13:28:28,221 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) HostStatsCollector
is running...
2013-08-09 13:28:28,702 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-1681129513:
Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer
} }
2013-08-09 13:28:28,775 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-1681129514:
Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer
} }
2013-08-09 13:28:29,045 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 
10.146.0.20 -- GET  command=queryAsyncJobResult&jobId=0c5003b9-e341-46c4-b58f-c9181a7ccba1&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049619735
2013-08-09 13:28:29,065 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) Async
job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ] completed
2013-08-09 13:28:29,075 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END===  10.146.0.20
-- GET  command=queryAsyncJobResult&jobId=0c5003b9-e341-46c4-b58f-c9181a7ccba1&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049619735

Attaching management server log file and cloud DB.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message