cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "edison su (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (CLOUDSTACK-5482) Vmware - When nfs was down for about 1 hour , when snapshots were in progress , snapshot job failed when nfs was brought up leaving behind snaphots in "CreatedOnPrimary" state.
Date Mon, 17 Nov 2014 23:41:33 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

edison su reassigned CLOUDSTACK-5482:
-------------------------------------

    Assignee: edison su  (was: Sateesh Chodapuneedi)

> Vmware - When nfs was down for about 1 hour , when snapshots were in progress , snapshot
job failed when nfs was brought up leaving behind  snaphots in "CreatedOnPrimary" state.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5482
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5482
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Build from 4.3
>            Reporter: Sangeetha Hariharan
>            Assignee: edison su
>             Fix For: 4.4.0, 4.5.0
>
>         Attachments: nfs12down.rar, vmware.rar, vmware.rar
>
>
> Set up :
> Advanced Zone with 2 5.1 ESXI hosts.
> Steps to reproduce the problem:
> 1. Deploy 5 Vms in each of the hosts , so we start with 11 Vms.
> 2. Start concurrent snapshots for ROOT volumes of all the Vms.
> 3. Shutdown the Secondary storage server when the snapshots are in the progress.
> 4. Bring the Secondary storage server up after 1 hour.
> When the secondary storage was down , 2 of the  snapshots were already completed. 5 of
them were in progress and the other 4 had not started yet.
> Once the secondary store was brought up , I see the snapshots that were in progress actually
continue to download to secondary and succeed. But the other 4 snapshots error out. 
> mysql> select volume_id,status,created from snapshots;
> +-----------+------------------+---------------------+
> | volume_id | status           | created             |
> +-----------+------------------+---------------------+
> |        22 | BackedUp         | 2013-12-12 23:24:13 |
> |        21 | Destroyed        | 2013-12-12 23:24:13 |
> |        20 | BackedUp         | 2013-12-12 23:24:14 |
> |        19 | Destroyed        | 2013-12-12 23:24:14 |
> |        18 | BackedUp         | 2013-12-12 23:24:14 |
> |        17 | BackedUp         | 2013-12-12 23:24:14 |
> |        16 | BackedUp         | 2013-12-12 23:24:14 |
> |        14 | BackedUp         | 2013-12-12 23:24:15 |
> |        25 | BackedUp         | 2013-12-12 23:24:15 |
> |        24 | BackedUp         | 2013-12-12 23:24:15 |
> |        23 | BackedUp         | 2013-12-12 23:24:15 |
> |        22 | CreatedOnPrimary | 2013-12-12 23:53:38 |
> |        21 | BackedUp         | 2013-12-12 23:53:38 |
> |        20 | BackedUp         | 2013-12-12 23:53:38 |
> |        19 | BackedUp         | 2013-12-12 23:53:39 |
> |        18 | CreatedOnPrimary | 2013-12-12 23:53:39 |
> |        17 | CreatedOnPrimary | 2013-12-12 23:53:40 |
> |        16 | CreatedOnPrimary | 2013-12-12 23:53:40 |
> |        14 | BackedUp         | 2013-12-12 23:53:40 |
> |        25 | BackedUp         | 2013-12-12 23:53:41 |
> |        24 | BackedUp         | 2013-12-12 23:53:41 |
> |        23 | BackedUp         | 2013-12-12 23:53:42 |
> |        21 | BackedUp         | 2013-12-13 00:53:37 |
> |        19 | BackedUp         | 2013-12-13 00:53:38 |
> +-----------+------------------+---------------------+
> 24 rows in set (0.00 sec)
> This leaves behind incomplete snapshots. The directory does not have a ovf file and has
incomplete vmdk file.
> [root@Rack3Host8 18]# ls -ltR
> .:
> total 12
> drwxr-xr-x. 2 root root 4096 Dec 12 22:56 36d7964c-e545-41d7-b303-96359a88dcef
> drwxr-xr-x. 2 root root 4096 Dec 12 22:30 68802f5f-84b1-42ad-8dca-4de7e83324e2
> ./36d7964c-e545-41d7-b303-96359a88dcef:
> total 403256
> -rw-r--r--. 1 root root 412524288 Dec 13 00:20 36d7964c-e545-41d7-b303-96359a88dcef-disk0.vmdk
> ./68802f5f-84b1-42ad-8dca-4de7e83324e2:
> total 448860
> -rw-r--r--. 1 root root 459168256 Dec 12 22:30 68802f5f-84b1-42ad-8dca-4de7e83324e2-disk0.vmdk
> -rw-r--r--. 1 root root      6454 Dec 12 22:30 68802f5f-84b1-42ad-8dca-4de7e83324e2.ovf
> [root@Rack3Host8 18]#
> Following exception seen in the management server logs:
> 2013-12-12 20:23:13,021 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 5-813367309:
Processing:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"backup
snapshot exception: Exception: java.lang.Exception\nMessage: Unable to finish the whole process
to package as a OVA file\n","wait":0}}] }
> 2013-12-12 20:23:13,022 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-83fb69a5 ctx-51e56052)
Seq 5-813367309: Received:  { Ans: , MgmtId: 95307354844397, via: 5, Ver: v1, Flags: 10, {
CopyCmdAnswer } }
> 2013-12-12 20:23:13,041 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-1:ctx-83fb69a5
ctx-51e56052) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception:
java.lang.Exception
> Message: Unable to finish the whole process to package as a OVA file
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy161.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1486)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy233.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message