Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D6E3710E63 for ; Fri, 6 Dec 2013 02:18:37 +0000 (UTC) Received: (qmail 46929 invoked by uid 500); 6 Dec 2013 02:18:37 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 46909 invoked by uid 500); 6 Dec 2013 02:18:37 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 46902 invoked by uid 500); 6 Dec 2013 02:18:37 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 46899 invoked by uid 99); 6 Dec 2013 02:18:37 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Dec 2013 02:18:37 +0000 Date: Fri, 6 Dec 2013 02:18:37 +0000 (UTC) From: "Sangeetha Hariharan (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (CLOUDSTACK-5395) When MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Sangeetha Hariharan created CLOUDSTACK-5395: ----------------------------------------------- Summary: When Key: CLOUDSTACK-5395 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5395 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Reporter: Sangeetha Hariharan Steps to reproduce the problem: Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms. We will be constantly writing to the ROOT volume. Change the backup.snapshot.wait to 10 mts and restart management server. Start concurrent snapshots for ROOT volumes of all the Vms. After 10 mts , the snapshots fail. They are present in the database in "CreatedOnPrimary" state. Vhd entries from the primary store fails to be cleaned up once the backsnapshpot job fails. Expected Behavior: We should be able to clean up the Vhd entries from the primary store when the backsnapshpot job fails. select * from snapshot_store_ref; 702 | 1 | 355 | 2013-12-06 01:25:43 | NULL | NULL | Primary | 0 | 0 | 0 | 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60 | Ready | 2 | 0 | 2013-12-06 01:25:44 | 81 | | 703 | 1 | 356 | 2013-12-06 01:25:43 | NULL | NULL | Primary | 0 | 0 | 0 | 9d88bc01-9406-41ad-a134-e74dc1457954 | Ready | 2 | 0 | 2013-12-06 01:26:12 | 80 | | 704 | 1 | 357 | 2013-12-06 01:25:43 | NULL | NULL | Primary | 0 | 0 | 0 | 2667f2bc-6086-4ec3-a88d-20811eabde91 | Ready | 2 | 0 | 2013-12-06 01:26:08 | 79 | | 705 | 1 | 358 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 522b2296-6960-46f2-af7d-10ddfbede1da | Ready | 2 | 0 | 2013-12-06 01:26:45 | 78 | | 706 | 1 | 359 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 3b94fa9d-a5a5-4441-8f9f-275dcef90368 | Ready | 2 | 0 | 2013-12-06 01:26:04 | 77 | | 707 | 1 | 360 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84 | Ready | 2 | 0 | 2013-12-06 01:25:59 | 76 | | 708 | 1 | 361 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 324e7552-b42a-4660-90d6-62015a7a478e | Ready | 2 | 0 | 2013-12-06 01:26:21 | 75 | | 709 | 1 | 362 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 65bd522c-c2c8-471a-be37-095558d058f2 | Ready | 2 | 0 | 2013-12-06 01:26:16 | 74 | | 710 | 1 | 363 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | d45ca6c7-7284-4150-907c-9499e9737c47 | Ready | 2 | 0 | 2013-12-06 01:25:46 | 73 | | 711 | 1 | 364 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 4422f362-0be5-4a10-b172-45678d56f807 | Ready | 2 | 0 | 2013-12-06 01:25:55 | 72 | | 712 | 1 | 365 | 2013-12-06 01:25:44 | NULL | NULL | Primary | 0 | 0 | 0 | 89ffd430-3c03-45d2-9c48-9384636b9cd8 | Ready | 2 | 0 | 2013-12-06 01:26:01 | 71 | | 714 | 1 | 366 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | fca5545c-9b83-4bc1-abd2-dd1bc82b23bd | Ready | 2 | 0 | 2013-12-06 01:25:53 | 70 | | 715 | 1 | 367 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | 033d8f55-8895-40b8-a120-11b28fa1f96e | Ready | 2 | 0 | 2013-12-06 01:25:50 | 69 | | 716 | 1 | 368 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | e4d02558-28c2-474e-a379-970b22f33f55 | Ready | 2 | 0 | 2013-12-06 01:26:23 | 68 | | 717 | 1 | 369 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | 6f7c1ca0-9877-44af-9f77-4db7b8efc934 | Ready | 2 | 0 | 2013-12-06 01:26:37 | 67 | | 718 | 1 | 370 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | e3860504-7e37-416a-8216-c4a021f5e8bf | Ready | 2 | 0 | 2013-12-06 01:25:47 | 66 | | 719 | 1 | 371 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | 48c2a9bd-e7ce-429f-be73-ef9c19964ad3 | Ready | 2 | 0 | 2013-12-06 01:26:17 | 65 | | 720 | 1 | 372 | 2013-12-06 01:25:45 | NULL | NULL | Primary | 0 | 0 | 0 | 3eabb081-d0d9-41d6-9c11-00e002031461 | Ready | 2 | 0 | 2013-12-06 01:25:51 | 64 | | 721 | 1 | 373 | 2013-12-06 01:25:46 | NULL | NULL | Primary | 0 | 0 | 0 | c9968fbb-0876-4846-94aa-e44ea7e3143d | Ready | 2 | 0 | 2013-12-06 01:26:06 | 63 | | 723 | 1 | 374 | 2013-12-06 01:25:46 | NULL | NULL | Primary | 0 | 0 | 0 | 72fdec4a-c2f1-48e1-80bf-806b6db46820 | Ready | 2 | 0 | 2013-12-06 01:26:02 | 62 | | 742 | 1 | 375 | 2013-12-06 01:40:43 | NULL | NULL | Primary | 0 | 0 | 0 | 8a692b5a-569e-473c-83c4-dc32c09c8491 | Ready | 2 | 0 | 2013-12-06 01:40:47 | 60 | +-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+ 2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy161.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy233.takeSnapshot(Unknown Source) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972) at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy161.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy233.takeSnapshot(Unknown Source) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) -- This message was sent by Atlassian JIRA (v6.1#6144)