cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangeetha Hariharan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-5262) Few of the snapshot creation from ROOT volume fails when there are concurrent snapshots in progress.
Date Wed, 27 Nov 2013 17:30:42 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sangeetha Hariharan updated CLOUDSTACK-5262:
--------------------------------------------

    Attachment: xen2.rar
                xen1-2.rar
                xen1.rar

> Few of  the snapshot creation from ROOT volume fails when there are concurrent snapshots
in progress.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5262
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5262
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.3.0
>         Environment: Build from 4.3
>            Reporter: Sangeetha Hariharan
>            Assignee: edison su
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: management-server.rar, xen1-2.rar, xen1.rar, xen2.rar
>
>
> Steps to reproduce the problem:
> Set up - Advanced zone with 2 Xenserver 6.2 hosts
> Deploy 10 Vms in each of the hosts , so we start with 20 Vms.
> We are constantly writing to the ROOT volume ( print timestamp every 1 minute).
> Start concurrent snapshots for ROOT volumes for all the Vms.
> Few of the snapshot jobs ( 4 of them) are failing due to the following exception:
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6)
Seq 2-756819795: Sending  { Cmd , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com),
Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
}
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6)
Seq 2-756819795: Executing:  { Cmd , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com),
Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
}
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-a502b0ee)
Seq 2-756819795: Executing request
> 2013-11-25 08:16:16,738 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-a502b0ee)
callHostPlugin failed for cmd: backupSnapshot with args secondaryStorageMountPath: 10.223.110.233:/export/home/sangeetha/felton/xen/secondary,
localMountPoint: /var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947, backupUuid: 07f55f61-e1a5-4880-a9a0-37e063e88e39,
path: snapshots/9/73, prevBackupUuid: snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71,
snapshotUuid: ee56cec0-632d-469f-9326-d683e9fc1425, isISCSI: false, primaryStorageSRUuid:
69ec4a37-f19c-8951-8b52-ebe3437f4b5e,  due to HandleInvalid clazz:task, handle:OpaqueRef:2d1621a0-7c9e-ea80-48ef-c7114ce3a63d
> 2013-11-25 08:16:16,740 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-a502b0ee)
unable to destroy task(com.xensource.xenapi.Task@4b81265b) on host(b8c15cde-f075-487b-86bc-9ef564900887)
due to
> You gave an invalid object reference.  The object may have recently been deleted.  The
class parameter gives the type of reference given, and the handle parameter echoes the bad
value given.
>         at com.xensource.xenapi.Types.checkResponse(Types.java:209)
>         at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
>         at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
>         at com.xensource.xenapi.Task.destroy(Task.java:616)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:4067)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1131)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 3-37675: Processing Seq 3-37675:  { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-11-25 08:16:17,100 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 3-37675: Sending Seq 3-37675:  { Ans: , MgmtId: 112516401760401, via: 3, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-11-25 08:16:17,755 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-364:ctx-a502b0ee)
Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e
to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> 2013-11-25 08:16:17,755 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-364:ctx-a502b0ee)
BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e
to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> com.cloud.utils.exception.CloudRuntimeException: Could not copy backupUuid: null from
primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary
due to null
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1160)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-a502b0ee)
Seq 2-756819795: Response Received:
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795:
Processing:  { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot
Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e
to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null","wait":0}}]
}
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6)
Seq 2-756819795: Received:  { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10,
{ CopyCmdAnswer } }
> 2013-11-25 08:16:17,768 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-165:ctx-bae63a1a
ctx-4f637cc6) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not
copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary
storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy160.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
>         at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy232.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,775 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-165:ctx-bae63a1a
ctx-4f637cc6) Take snapshot: 73 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
>         at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy160.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
>         at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
>         at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy232.takeSnapshot(Unknown Source)
>         at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due
to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e
to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
>         at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
>         at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
>         at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
>         ... 42 more
> 2013-11-25 08:16:17,776 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-165:ctx-bae63a1a)
Complete async job-631, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 73"}
> mysql> select  job_cmd , account_id , job_result, job_result_code, job_status  from
async_job where job_status=2 and  job_cmd="org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd";
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | job_cmd                                                           | account_id | job_result
                                                                                         
                                                                                      | job_result_code
| job_status |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |          9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 73"} |          
  530 |          2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |          9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 80"} |          
  530 |          2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |          9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 83"} |          
  530 |          2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |          9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume 85"} |          
  530 |          2 |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> 4 rows in set (0.01 sec)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message