cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangeetha Hariharan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-5395) When backup snapshot fails becasue of backup.snapshot.wait time exceeding , the vhd entries form the primary store is not getting cleared.
Date Fri, 06 Dec 2013 21:39:35 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sangeetha Hariharan updated CLOUDSTACK-5395:
--------------------------------------------

    Description: 
Steps to reproduce the problem:

Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms.
We will be constantly writing to the ROOT volume.

Change the backup.snapshot.wait to 10 mts and restart management server.

Start concurrent snapshots for ROOT volumes of all the Vms.

After 10 mts ,  the snapshots fail. They are present in the database in "CreatedOnPrimary" state.

Vhd entries from the primary store fails to be cleaned up once the backsnapshpot job fails.

In this case  ,API fails with - "Failed to create snapshot due to an internal error creating snapshot for volume 81" . We should be able to present a more meaningful error message in this case.

Expected Behavior:
We should be able to clean up the  Vhd entries from the primary store when the backsnapshpot job fails.

In such cases , instead of 
select * from snapshot_store_ref;

 702 |        1 |         355 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60                | Ready     |            2 |       0 | 2013-12-06 01:25:44 |        81 |
| 703 |        1 |         356 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 9d88bc01-9406-41ad-a134-e74dc1457954                | Ready     |            2 |       0 | 2013-12-06 01:26:12 |        80 |
| 704 |        1 |         357 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2667f2bc-6086-4ec3-a88d-20811eabde91                | Ready     |            2 |       0 | 2013-12-06 01:26:08 |        79 |
| 705 |        1 |         358 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 522b2296-6960-46f2-af7d-10ddfbede1da                | Ready     |            2 |       0 | 2013-12-06 01:26:45 |        78 |
| 706 |        1 |         359 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3b94fa9d-a5a5-4441-8f9f-275dcef90368                | Ready     |            2 |       0 | 2013-12-06 01:26:04 |        77 |
| 707 |        1 |         360 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84                | Ready     |            2 |       0 | 2013-12-06 01:25:59 |        76 |
| 708 |        1 |         361 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 324e7552-b42a-4660-90d6-62015a7a478e                | Ready     |            2 |       0 | 2013-12-06 01:26:21 |        75 |
| 709 |        1 |         362 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 65bd522c-c2c8-471a-be37-095558d058f2                | Ready     |            2 |       0 | 2013-12-06 01:26:16 |        74 |
| 710 |        1 |         363 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | d45ca6c7-7284-4150-907c-9499e9737c47                | Ready     |            2 |       0 | 2013-12-06 01:25:46 |        73 |
| 711 |        1 |         364 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 4422f362-0be5-4a10-b172-45678d56f807                | Ready     |            2 |       0 | 2013-12-06 01:25:55 |        72 |
| 712 |        1 |         365 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 89ffd430-3c03-45d2-9c48-9384636b9cd8                | Ready     |            2 |       0 | 2013-12-06 01:26:01 |        71 |
| 714 |        1 |         366 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | fca5545c-9b83-4bc1-abd2-dd1bc82b23bd                | Ready     |            2 |       0 | 2013-12-06 01:25:53 |        70 |
| 715 |        1 |         367 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 033d8f55-8895-40b8-a120-11b28fa1f96e                | Ready     |            2 |       0 | 2013-12-06 01:25:50 |        69 |
| 716 |        1 |         368 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e4d02558-28c2-474e-a379-970b22f33f55                | Ready     |            2 |       0 | 2013-12-06 01:26:23 |        68 |
| 717 |        1 |         369 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 6f7c1ca0-9877-44af-9f77-4db7b8efc934                | Ready     |            2 |       0 | 2013-12-06 01:26:37 |        67 |
| 718 |        1 |         370 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e3860504-7e37-416a-8216-c4a021f5e8bf                | Ready     |            2 |       0 | 2013-12-06 01:25:47 |        66 |
| 719 |        1 |         371 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 48c2a9bd-e7ce-429f-be73-ef9c19964ad3                | Ready     |            2 |       0 | 2013-12-06 01:26:17 |        65 |
| 720 |        1 |         372 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3eabb081-d0d9-41d6-9c11-00e002031461                | Ready     |            2 |       0 | 2013-12-06 01:25:51 |        64 |
| 721 |        1 |         373 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | c9968fbb-0876-4846-94aa-e44ea7e3143d                | Ready     |            2 |       0 | 2013-12-06 01:26:06 |        63 |
| 723 |        1 |         374 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 72fdec4a-c2f1-48e1-80bf-806b6db46820                | Ready     |            2 |       0 | 2013-12-06 01:26:02 |        62 |
| 742 |        1 |         375 | 2013-12-06 01:40:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 8a692b5a-569e-473c-83c4-dc32c09c8491                | Ready     |            2 |       0 | 2013-12-06 01:40:47 |        60 |
+-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+



2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
        at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
        at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2013-12-05 20:35:50,550 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-22:ctx-ae0b13d4 ctx-1bc38e81) Take snapshot: 81 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@db9844a9
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
        ... 44 more
2013-12-05 20:35:50,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Complete async job-697, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 81"}
2013-12-05 20:35:50,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-697




  was:
Steps to reproduce the problem:

Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms.
We will be constantly writing to the ROOT volume.

Change the backup.snapshot.wait to 10 mts and restart management server.

Start concurrent snapshots for ROOT volumes of all the Vms.

After 10 mts ,  the snapshots fail. They are present in the database in "CreatedOnPrimary" state.

Vhd entries from the primary store fails to be cleaned up once the backsnapshpot job fails.

Expected Behavior:
We should be able to clean up the  Vhd entries from the primary store when the backsnapshpot job fails.

In such cases , instead of 
select * from snapshot_store_ref;

 702 |        1 |         355 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60                | Ready     |            2 |       0 | 2013-12-06 01:25:44 |        81 |
| 703 |        1 |         356 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 9d88bc01-9406-41ad-a134-e74dc1457954                | Ready     |            2 |       0 | 2013-12-06 01:26:12 |        80 |
| 704 |        1 |         357 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2667f2bc-6086-4ec3-a88d-20811eabde91                | Ready     |            2 |       0 | 2013-12-06 01:26:08 |        79 |
| 705 |        1 |         358 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 522b2296-6960-46f2-af7d-10ddfbede1da                | Ready     |            2 |       0 | 2013-12-06 01:26:45 |        78 |
| 706 |        1 |         359 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3b94fa9d-a5a5-4441-8f9f-275dcef90368                | Ready     |            2 |       0 | 2013-12-06 01:26:04 |        77 |
| 707 |        1 |         360 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84                | Ready     |            2 |       0 | 2013-12-06 01:25:59 |        76 |
| 708 |        1 |         361 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 324e7552-b42a-4660-90d6-62015a7a478e                | Ready     |            2 |       0 | 2013-12-06 01:26:21 |        75 |
| 709 |        1 |         362 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 65bd522c-c2c8-471a-be37-095558d058f2                | Ready     |            2 |       0 | 2013-12-06 01:26:16 |        74 |
| 710 |        1 |         363 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | d45ca6c7-7284-4150-907c-9499e9737c47                | Ready     |            2 |       0 | 2013-12-06 01:25:46 |        73 |
| 711 |        1 |         364 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 4422f362-0be5-4a10-b172-45678d56f807                | Ready     |            2 |       0 | 2013-12-06 01:25:55 |        72 |
| 712 |        1 |         365 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 89ffd430-3c03-45d2-9c48-9384636b9cd8                | Ready     |            2 |       0 | 2013-12-06 01:26:01 |        71 |
| 714 |        1 |         366 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | fca5545c-9b83-4bc1-abd2-dd1bc82b23bd                | Ready     |            2 |       0 | 2013-12-06 01:25:53 |        70 |
| 715 |        1 |         367 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 033d8f55-8895-40b8-a120-11b28fa1f96e                | Ready     |            2 |       0 | 2013-12-06 01:25:50 |        69 |
| 716 |        1 |         368 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e4d02558-28c2-474e-a379-970b22f33f55                | Ready     |            2 |       0 | 2013-12-06 01:26:23 |        68 |
| 717 |        1 |         369 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 6f7c1ca0-9877-44af-9f77-4db7b8efc934                | Ready     |            2 |       0 | 2013-12-06 01:26:37 |        67 |
| 718 |        1 |         370 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e3860504-7e37-416a-8216-c4a021f5e8bf                | Ready     |            2 |       0 | 2013-12-06 01:25:47 |        66 |
| 719 |        1 |         371 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 48c2a9bd-e7ce-429f-be73-ef9c19964ad3                | Ready     |            2 |       0 | 2013-12-06 01:26:17 |        65 |
| 720 |        1 |         372 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3eabb081-d0d9-41d6-9c11-00e002031461                | Ready     |            2 |       0 | 2013-12-06 01:25:51 |        64 |
| 721 |        1 |         373 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | c9968fbb-0876-4846-94aa-e44ea7e3143d                | Ready     |            2 |       0 | 2013-12-06 01:26:06 |        63 |
| 723 |        1 |         374 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 72fdec4a-c2f1-48e1-80bf-806b6db46820                | Ready     |            2 |       0 | 2013-12-06 01:26:02 |        62 |
| 742 |        1 |         375 | 2013-12-06 01:40:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 8a692b5a-569e-473c-83c4-dc32c09c8491                | Ready     |            2 |       0 | 2013-12-06 01:40:47 |        60 |
+-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+



2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
        at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
        at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2013-12-05 20:35:50,550 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-22:ctx-ae0b13d4 ctx-1bc38e81) Take snapshot: 81 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@db9844a9
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
        ... 44 more
2013-12-05 20:35:50,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Complete async job-697, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 81"}
2013-12-05 20:35:50,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-697





> When backup snapshot fails becasue of backup.snapshot.wait time exceeding , the vhd entries form the primary store is not getting cleared.
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5395
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5395
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Build from 4.3
>            Reporter: Sangeetha Hariharan
>            Assignee: edison su
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: 1205logs.rar
>
>
> Steps to reproduce the problem:
> Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms.
> We will be constantly writing to the ROOT volume.
> Change the backup.snapshot.wait to 10 mts and restart management server.
> Start concurrent snapshots for ROOT volumes of all the Vms.
> After 10 mts ,  the snapshots fail. They are present in the database in "CreatedOnPrimary" state.
> Vhd entries from the primary store fails to be cleaned up once the backsnapshpot job fails.
> In this case  ,API fails with - "Failed to create snapshot due to an internal error creating snapshot for volume 81" . We should be able to present a more meaningful error message in this case.
> Expected Behavior:
> We should be able to clean up the  Vhd entries from the primary store when the backsnapshpot job fails.
> In such cases , instead of 
> select * from snapshot_store_ref;
>  702 |        1 |         355 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60                | Ready     |            2 |       0 | 2013-12-06 01:25:44 |        81 |
> | 703 |        1 |         356 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 9d88bc01-9406-41ad-a134-e74dc1457954                | Ready     |            2 |       0 | 2013-12-06 01:26:12 |        80 |
> | 704 |        1 |         357 | 2013-12-06 01:25:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 2667f2bc-6086-4ec3-a88d-20811eabde91                | Ready     |            2 |       0 | 2013-12-06 01:26:08 |        79 |
> | 705 |        1 |         358 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 522b2296-6960-46f2-af7d-10ddfbede1da                | Ready     |            2 |       0 | 2013-12-06 01:26:45 |        78 |
> | 706 |        1 |         359 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3b94fa9d-a5a5-4441-8f9f-275dcef90368                | Ready     |            2 |       0 | 2013-12-06 01:26:04 |        77 |
> | 707 |        1 |         360 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84                | Ready     |            2 |       0 | 2013-12-06 01:25:59 |        76 |
> | 708 |        1 |         361 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 324e7552-b42a-4660-90d6-62015a7a478e                | Ready     |            2 |       0 | 2013-12-06 01:26:21 |        75 |
> | 709 |        1 |         362 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 65bd522c-c2c8-471a-be37-095558d058f2                | Ready     |            2 |       0 | 2013-12-06 01:26:16 |        74 |
> | 710 |        1 |         363 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | d45ca6c7-7284-4150-907c-9499e9737c47                | Ready     |            2 |       0 | 2013-12-06 01:25:46 |        73 |
> | 711 |        1 |         364 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 4422f362-0be5-4a10-b172-45678d56f807                | Ready     |            2 |       0 | 2013-12-06 01:25:55 |        72 |
> | 712 |        1 |         365 | 2013-12-06 01:25:44 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 89ffd430-3c03-45d2-9c48-9384636b9cd8                | Ready     |            2 |       0 | 2013-12-06 01:26:01 |        71 |
> | 714 |        1 |         366 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | fca5545c-9b83-4bc1-abd2-dd1bc82b23bd                | Ready     |            2 |       0 | 2013-12-06 01:25:53 |        70 |
> | 715 |        1 |         367 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 033d8f55-8895-40b8-a120-11b28fa1f96e                | Ready     |            2 |       0 | 2013-12-06 01:25:50 |        69 |
> | 716 |        1 |         368 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e4d02558-28c2-474e-a379-970b22f33f55                | Ready     |            2 |       0 | 2013-12-06 01:26:23 |        68 |
> | 717 |        1 |         369 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 6f7c1ca0-9877-44af-9f77-4db7b8efc934                | Ready     |            2 |       0 | 2013-12-06 01:26:37 |        67 |
> | 718 |        1 |         370 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | e3860504-7e37-416a-8216-c4a021f5e8bf                | Ready     |            2 |       0 | 2013-12-06 01:25:47 |        66 |
> | 719 |        1 |         371 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 48c2a9bd-e7ce-429f-be73-ef9c19964ad3                | Ready     |            2 |       0 | 2013-12-06 01:26:17 |        65 |
> | 720 |        1 |         372 | 2013-12-06 01:25:45 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 3eabb081-d0d9-41d6-9c11-00e002031461                | Ready     |            2 |       0 | 2013-12-06 01:25:51 |        64 |
> | 721 |        1 |         373 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | c9968fbb-0876-4846-94aa-e44ea7e3143d                | Ready     |            2 |       0 | 2013-12-06 01:26:06 |        63 |
> | 723 |        1 |         374 | 2013-12-06 01:25:46 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 72fdec4a-c2f1-48e1-80bf-806b6db46820                | Ready     |            2 |       0 | 2013-12-06 01:26:02 |        62 |
> | 742 |        1 |         375 | 2013-12-06 01:40:43 | NULL         | NULL   | Primary    |    0 |             0 |                  0 | 8a692b5a-569e-473c-83c4-dc32c09c8491                | Ready     |            2 |       0 | 2013-12-06 01:40:47 |        60 |
> +-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+
> 2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
>         at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
>         at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
>         at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-12-05 20:35:50,550 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-22:ctx-ae0b13d4 ctx-1bc38e81) Take snapshot: 81 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@db9844a9
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         ... 44 more
> 2013-12-05 20:35:50,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Complete async job-697, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 81"}
> 2013-12-05 20:35:50,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-697



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message