cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangeetha Hariharan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-5482) Vmware - When nfs was down for about 1 hour , when snapshots were in progress , snapshot job failed when nfs was brought up leaving behind snaphots in "CreatedOnPrimary" state.
Date Fri, 13 Dec 2013 01:54:07 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sangeetha Hariharan updated CLOUDSTACK-5482:
--------------------------------------------

    Description: 
Set up :
Advanced Zone with 2 5.1 ESXI hosts.

Steps to reproduce the problem:

1. Deploy 5 Vms in each of the hosts , so we start with 11 Vms.
2. Start concurrent snapshots for ROOT volumes of all the Vms.
3. Shutdown the Secondary storage server when the snapshots are in the progress.
4. Bring the Secondary storage server up after 1 hour.

When the secondary storage was down , 2 of the  snapshots were already completed. 5 of them
were in progress and the other 4 had not started yet.

Once the secondary store was brought up , I see the snapshots that were in progress actually
continue to download to secondary and succeed. But the other 4 snapshots error out. 

mysql> select volume_id,status,created from snapshots;
+-----------+------------------+---------------------+
| volume_id | status           | created             |
+-----------+------------------+---------------------+
|        22 | BackedUp         | 2013-12-12 23:24:13 |
|        21 | Destroyed        | 2013-12-12 23:24:13 |
|        20 | BackedUp         | 2013-12-12 23:24:14 |
|        19 | Destroyed        | 2013-12-12 23:24:14 |
|        18 | BackedUp         | 2013-12-12 23:24:14 |
|        17 | BackedUp         | 2013-12-12 23:24:14 |
|        16 | BackedUp         | 2013-12-12 23:24:14 |
|        14 | BackedUp         | 2013-12-12 23:24:15 |
|        25 | BackedUp         | 2013-12-12 23:24:15 |
|        24 | BackedUp         | 2013-12-12 23:24:15 |
|        23 | BackedUp         | 2013-12-12 23:24:15 |
|        22 | CreatedOnPrimary | 2013-12-12 23:53:38 |
|        21 | BackedUp         | 2013-12-12 23:53:38 |
|        20 | BackedUp         | 2013-12-12 23:53:38 |
|        19 | BackedUp         | 2013-12-12 23:53:39 |
|        18 | CreatedOnPrimary | 2013-12-12 23:53:39 |
|        17 | CreatedOnPrimary | 2013-12-12 23:53:40 |
|        16 | CreatedOnPrimary | 2013-12-12 23:53:40 |
|        14 | BackedUp         | 2013-12-12 23:53:40 |
|        25 | BackedUp         | 2013-12-12 23:53:41 |
|        24 | BackedUp         | 2013-12-12 23:53:41 |
|        23 | BackedUp         | 2013-12-12 23:53:42 |
|        21 | BackedUp         | 2013-12-13 00:53:37 |
|        19 | BackedUp         | 2013-12-13 00:53:38 |
+-----------+------------------+---------------------+
24 rows in set (0.00 sec)

Following exception seen in the management server logs:

2013-12-12 20:23:13,021 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 5-813367309:
Processing:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"backup
snapshot exception: Exception: java.lang.Exception\nMessage: Unable to finish the whole process
to package as a OVA file\n","wait":0}}] }
2013-12-12 20:23:13,022 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-83fb69a5 ctx-51e56052)
Seq 5-813367309: Received:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, {
CopyCmdAnswer } }
2013-12-12 20:23:13,041 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-1:ctx-83fb69a5 ctx-51e56052)
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception: java.lang.Exception
Message: Unable to finish the whole process to package as a OVA file

        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy161.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1486)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy233.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
        at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
        at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)




  was:
Set up :
Advanced Zone with 2 5.1 ESXI hosts.

Steps to reproduce the problem:

1. Deploy 5 Vms in each of the hosts , so we start with 11 Vms.
2. Start concurrent snapshots for ROOT volumes of all the Vms.
3. Shutdown the Secondary storage server when the snapshots are in the progress.
4. Bring the Secondary storage server up after 1 hour.

When the secondary storage was down , 2 of the  snapshots were already completed. 5 of them
were in progress and the other 4 had not started yet.

Once the secondary store was brought up , I see the snapshots that were in progress actually
continue to download to secondary and succeed. But the other 4 snapshots error out. 

mysql> select volume_id,status,created from snapshots;
+-----------+------------------+---------------------+
| volume_id | status           | created             |
+-----------+------------------+---------------------+
|        22 | BackedUp         | 2013-12-12 23:24:13 |
|        21 | Destroyed        | 2013-12-12 23:24:13 |
|        20 | BackedUp         | 2013-12-12 23:24:14 |
|        19 | Destroyed        | 2013-12-12 23:24:14 |
|        18 | BackedUp         | 2013-12-12 23:24:14 |
|        17 | BackedUp         | 2013-12-12 23:24:14 |
|        16 | BackedUp         | 2013-12-12 23:24:14 |
|        14 | BackedUp         | 2013-12-12 23:24:15 |
|        25 | BackedUp         | 2013-12-12 23:24:15 |
|        24 | BackedUp         | 2013-12-12 23:24:15 |
|        23 | BackedUp         | 2013-12-12 23:24:15 |
|        22 | CreatedOnPrimary | 2013-12-12 23:53:38 |
|        21 | BackedUp         | 2013-12-12 23:53:38 |
|        20 | BackedUp         | 2013-12-12 23:53:38 |
|        19 | BackedUp         | 2013-12-12 23:53:39 |
|        18 | CreatedOnPrimary | 2013-12-12 23:53:39 |
|        17 | CreatedOnPrimary | 2013-12-12 23:53:40 |
|        16 | CreatedOnPrimary | 2013-12-12 23:53:40 |
|        14 | BackedUp         | 2013-12-12 23:53:40 |
|        25 | BackedUp         | 2013-12-12 23:53:41 |
|        24 | BackedUp         | 2013-12-12 23:53:41 |
|        23 | BackedUp         | 2013-12-12 23:53:42 |
|        21 | BackedUp         | 2013-12-13 00:53:37 |
|        19 | BackedUp         | 2013-12-13 00:53:38 |
+-----------+------------------+---------------------+
24 rows in set (0.00 sec)





> Vmware - When nfs was down for about 1 hour , when snapshots were in progress , snapshot
job failed when nfs was brought up leaving behind  snaphots in "CreatedOnPrimary" state.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5482
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5482
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Build from 4.3
>            Reporter: Sangeetha Hariharan
>             Fix For: 4.3.0
>
>
> Set up :
> Advanced Zone with 2 5.1 ESXI hosts.
> Steps to reproduce the problem:
> 1. Deploy 5 Vms in each of the hosts , so we start with 11 Vms.
> 2. Start concurrent snapshots for ROOT volumes of all the Vms.
> 3. Shutdown the Secondary storage server when the snapshots are in the progress.
> 4. Bring the Secondary storage server up after 1 hour.
> When the secondary storage was down , 2 of the  snapshots were already completed. 5 of
them were in progress and the other 4 had not started yet.
> Once the secondary store was brought up , I see the snapshots that were in progress actually
continue to download to secondary and succeed. But the other 4 snapshots error out. 
> mysql> select volume_id,status,created from snapshots;
> +-----------+------------------+---------------------+
> | volume_id | status           | created             |
> +-----------+------------------+---------------------+
> |        22 | BackedUp         | 2013-12-12 23:24:13 |
> |        21 | Destroyed        | 2013-12-12 23:24:13 |
> |        20 | BackedUp         | 2013-12-12 23:24:14 |
> |        19 | Destroyed        | 2013-12-12 23:24:14 |
> |        18 | BackedUp         | 2013-12-12 23:24:14 |
> |        17 | BackedUp         | 2013-12-12 23:24:14 |
> |        16 | BackedUp         | 2013-12-12 23:24:14 |
> |        14 | BackedUp         | 2013-12-12 23:24:15 |
> |        25 | BackedUp         | 2013-12-12 23:24:15 |
> |        24 | BackedUp         | 2013-12-12 23:24:15 |
> |        23 | BackedUp         | 2013-12-12 23:24:15 |
> |        22 | CreatedOnPrimary | 2013-12-12 23:53:38 |
> |        21 | BackedUp         | 2013-12-12 23:53:38 |
> |        20 | BackedUp         | 2013-12-12 23:53:38 |
> |        19 | BackedUp         | 2013-12-12 23:53:39 |
> |        18 | CreatedOnPrimary | 2013-12-12 23:53:39 |
> |        17 | CreatedOnPrimary | 2013-12-12 23:53:40 |
> |        16 | CreatedOnPrimary | 2013-12-12 23:53:40 |
> |        14 | BackedUp         | 2013-12-12 23:53:40 |
> |        25 | BackedUp         | 2013-12-12 23:53:41 |
> |        24 | BackedUp         | 2013-12-12 23:53:41 |
> |        23 | BackedUp         | 2013-12-12 23:53:42 |
> |        21 | BackedUp         | 2013-12-13 00:53:37 |
> |        19 | BackedUp         | 2013-12-13 00:53:38 |
> +-----------+------------------+---------------------+
> 24 rows in set (0.00 sec)
> Following exception seen in the management server logs:
> 2013-12-12 20:23:13,021 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 5-813367309:
Processing:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"backup
snapshot exception: Exception: java.lang.Exception\nMessage: Unable to finish the whole process
to package as a OVA file\n","wait":0}}] }
> 2013-12-12 20:23:13,022 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-83fb69a5 ctx-51e56052)
Seq 5-813367309: Received:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, {
CopyCmdAnswer } }
> 2013-12-12 20:23:13,041 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-1:ctx-83fb69a5
ctx-51e56052) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception:
java.lang.Exception
> Message: Unable to finish the whole process to package as a OVA file
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1486)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message