cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjeev N (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-2583) [Object_Store_Refactor] Failed to create snapshot from root disk of a guest VM
Date Tue, 09 Jul 2013 09:43:49 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13703092#comment-13703092
] 

Sanjeev N commented on CLOUDSTACK-2583:
---------------------------------------

Verified this issue on the latest master build. This time I din't see any issues with s3xen
persent on xenserver.
Snapshot creation was successful, however copying to s3 image store failed. Following is the
exception from the management server log file:

2013-07-09 10:53:01,518 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10
2013-07-09 10:53:01,582 INFO  [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10) VOLSS:
createSnapshotCmd starts:1373381581582
2013-07-09 10:53:01,685 DEBUG [agent.transport.Request] (Job-Executor-10:job-10) Seq 1-1762853387:
Sending  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"97e02ab2-c42f-42cc-816a-570dac81d6c8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-4","size":21474836480,"path":"28677700-7303-4015-a386-e55fb0f0ac89","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"VHD","id":4},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"vmName":"i-2-4-VM","name":"vm1_ROOT-4_20130709145301","hypervisorType":"XenServer","id":2}},"wait":0}}]
}
2013-07-09 10:53:01,687 DEBUG [agent.transport.Request] (Job-Executor-10:job-10) Seq 1-1762853387:
Executing:  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"97e02ab2-c42f-42cc-816a-570dac81d6c8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-4","size":21474836480,"path":"28677700-7303-4015-a386-e55fb0f0ac89","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"VHD","id":4},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"vmName":"i-2-4-VM","name":"vm1_ROOT-4_20130709145301","hypervisorType":"XenServer","id":2}},"wait":0}}]
}
2013-07-09 10:53:01,687 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq
1-1762853387: Executing request
2013-07-09 10:53:03,029 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq
1-1762853387: Response Received:
2013-07-09 10:53:03,030 DEBUG [agent.transport.Request] (DirectAgent-163:null) Seq 1-1762853387:
Processing:  { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"a5b44a59-f4d1-4ea1-ba76-5513cb595daa","id":0}},"result":true,"wait":0}}]
}
2013-07-09 10:53:03,030 DEBUG [agent.transport.Request] (Job-Executor-10:job-10) Seq 1-1762853387:
Received:  { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer
} }
2013-07-09 10:53:03,154 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10)
copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2013-07-09 10:53:03,201 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10)
needCacheStorage true, dest at snapshots/2/4 dest role Imagea5b44a59-f4d1-4ea1-ba76-5513cb595daa
src role Primary
2013-07-09 10:53:03,219 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10)
copy snasphot failed: java.lang.NullPointerException
2013-07-09 10:53:03,219 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10)
copy failed
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copySnapshot(AncientDataMotionStrategy.java:417)
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:349)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:267)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:127)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:197)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:981)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1171)
        at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2663)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:514)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-07-09 10:53:03,240 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-10)
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:282)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:127)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:197)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:981)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1171)
        at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2663)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:514)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-07-09 10:53:03,259 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-10)
Take snapshot: 4 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to create
snapshot
2013-07-09 10:53:03,270 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10) Complete
async job-10, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to
create snapshot due to an internal error creating snapshot for volume 4

I could see the snapshot file 28677700-7303-4015-a386-e55fb0f0ac89.vhd on my primary storage.


mysql> select * from snapshots;
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
| id | data_center_id | account_id | domain_id | volume_id | disk_offering_id | status   
       | path | name                      | uuid                                 | snapshot_type
| type_description | size        | created             | removed | backup_snap_id | swift_id
| sechost_id | prev_snap_id | hypervisor_type | version | s3_id |
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
|  1 |              1 |          2 |         1 |         4 |                1 | CreatedOnPrimary
| NULL | vm1_ROOT-4_20130709144517 | 4e435bc7-9b10-49e9-a1a7-711d29537df3 |             0
| MANUAL           | 21474836480 | 2013-07-09 14:45:17 | NULL    | NULL           |     NULL
|       NULL |         NULL | XenServer       | 2.2     |  NULL |
|  2 |              1 |          2 |         1 |         4 |                1 | CreatedOnPrimary
| NULL | vm1_ROOT-4_20130709145301 | 9bd6a928-c409-4ca5-841e-8e4be510fdda |             0
| MANUAL           | 21474836480 | 2013-07-09 14:53:01 | NULL    | NULL           |     NULL
|       NULL |         NULL | XenServer       | 2.2     |  NULL |
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
2 rows in set (0.01 sec)

Also I dint see any erros/failures in SMLog in xenserver during snapshot creation.


                
> [Object_Store_Refactor] Failed to create snapshot from root disk of a guest VM
> ------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-2583
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2583
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Snapshot
>    Affects Versions: 4.2.0
>         Environment: Build from object_store feature branch
>            Reporter: Sanjeev N
>            Assignee: John Burwell
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: management-server.rar
>
>
> [Object_Store_Refactor] Failed to create snapshot from root disk of a guest VM
> Steps to Reproduce:
> =================
> 1.Bring up CS in advanced zone with S3 as the secondary storage provider
> 2.Deploy guest with with the default centos template template downloaded
> 3.Try to take snapshot on the root disk of the guest vm
> Observations:
> ======2013-05-20 11:51:21,930 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===START===
 10.146.0.15 -- GET  command=createSnapshot&volumeid=135a3df1-20e1-4b4b-a433-86976ddc9218&response=json&sessionkey=YUyQ2NNnnp%2FGwC0zjw71QxChfHQ%3D&_=1369045371277
> 2013-05-20 11:51:22,046 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-281:null)
Seq 1-774570942: Response Received:
> 2013-05-20 11:51:22,046 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-774570942:
Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer
} }
> 2013-05-20 11:51:22,124 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-13:null)
submit async job-17, details: AsyncJobVO {id:17, userId: 2, accountId: 2, sessionKey: null,
instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"YUyQ2NNnnp/GwC0zjw71QxChfHQ\u003d","ctxUserId":"2","httpmethod":"GET","volumeid":"135a3df1-20e1-4b4b-a433-86976ddc9218","_":"1369045371277","ctxAccountId":"2","ctxStartEventId":"70"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
> 2013-05-20 11:51:22,127 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===END===
 10.146.0.15 -- GET  command=createSnapshot&volumeid=135a3df1-20e1-4b4b-a433-86976ddc9218&response=json&sessionkey=YUyQ2NNnnp%2FGwC0zjw71QxChfHQ%3D&_=1369045371277
> 2013-05-20 11:51:22,131 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-4:job-17)
Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-17
> 2013-05-20 11:51:22,188 INFO  [user.snapshot.CreateSnapshotCmd] (Job-Executor-4:job-17)
VOLSS: createSnapshotCmd starts:1369065082188
> 2013-05-20 11:51:22,300 DEBUG [agent.transport.Request] (Job-Executor-4:job-17) Seq 1-774570943:
Sending  { Cmd , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"135a3df1-20e1-4b4b-a433-86976ddc9218","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-3","size":21474836480,"path":"3fa9a666-20da-4adc-af07-46c92ac1b8f1","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"vmName":"i-2-3-VM","name":"vm1_ROOT-3_20130520155121","hypervisorType":"XenServer","id":1}},"wait":0}}]
}
> 2013-05-20 11:51:22,301 DEBUG [agent.transport.Request] (Job-Executor-4:job-17) Seq 1-774570943:
Executing:  { Cmd , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"135a3df1-20e1-4b4b-a433-86976ddc9218","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-3","size":21474836480,"path":"3fa9a666-20da-4adc-af07-46c92ac1b8f1","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"vmName":"i-2-3-VM","name":"vm1_ROOT-3_20130520155121","hypervisorType":"XenServer","id":1}},"wait":0}}]
}
> 2013-05-20 11:51:22,301 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-290:null)
Seq 1-774570943: Executing request
> 2013-05-20 11:51:24,090 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-290:null)
Seq 1-774570943: Response Received:
> 2013-05-20 11:51:24,091 DEBUG [agent.transport.Request] (DirectAgent-290:null) Seq 1-774570943:
Processing:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"021a0adf-2524-4d15-8cf6-1a09b377fd5c","id":0}},"result":true,"wait":0}}]
}
> 2013-05-20 11:51:24,091 DEBUG [agent.transport.Request] (Job-Executor-4:job-17) Seq 1-774570943:
Received:  { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer
} }
> 2013-05-20 11:51:24,184 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-4:job-17)
Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: Can't find cache storage in zone: null
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:303)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:134)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:208)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1003)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1031)
>         at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2601)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-05-20 11:51:24,201 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-4:job-17)
Complete async job-17, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text:
Failed to create snapshot due to an internal error creating snapshot for volume 3
> 2.Volume state from volumes table:
> mysql> select * from volumes where id=3\G;
> *************************** 1. row ***************************
>                         id: 3
>                 account_id: 2
>                  domain_id: 1
>                    pool_id: 1
>               last_pool_id: NULL
>                instance_id: 3
>                  device_id: 0
>                       name: ROOT-3
>                       uuid: 135a3df1-20e1-4b4b-a433-86976ddc9218
>                       size: 21474836480
>                     folder: NULL
>                       path: 3fa9a666-20da-4adc-af07-46c92ac1b8f1
>                     pod_id: NULL
>             data_center_id: 1
>                 iscsi_name: NULL
>                    host_ip: NULL
>                volume_type: ROOT
>                  pool_type: NULL
>           disk_offering_id: 1
>                template_id: 5
> first_snapshot_backup_uuid: NULL
>                recreatable: 0
>                    created: 2013-05-20 10:20:34
>                   attached: NULL
>                    updated: 2013-05-20 15:51:24
>                    removed: NULL
>                      state: Ready
>                 chain_info: NULL
>               update_count: 4
>                  disk_type: NULL
>             display_volume: 0
> 1 row in set (0.00 sec)
> ERROR:
> No query specified

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message