Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E201C10657 for ; Sun, 20 Oct 2013 17:08:53 +0000 (UTC) Received: (qmail 27722 invoked by uid 500); 20 Oct 2013 17:08:52 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 27540 invoked by uid 500); 20 Oct 2013 17:08:46 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 27527 invoked by uid 500); 20 Oct 2013 17:08:43 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 27522 invoked by uid 99); 20 Oct 2013 17:08:41 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Oct 2013 17:08:41 +0000 Date: Sun, 20 Oct 2013 17:08:41 +0000 (UTC) From: "le cong duan (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (CLOUDSTACK-4902) Fail to create snapshot with KVM when run multiple Hosts in Cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 le cong duan created CLOUDSTACK-4902: ---------------------------------------- Summary: Fail to create snapshot with KVM when run multiple Hosts in Cluster Key: CLOUDSTACK-4902 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: KVM, Management Server Affects Versions: 4.2.0 Environment: Managegemnet server: CentOS 6.3 64bit KVM Host: Ubuntu 12.04.1 64bit Reporter: le cong duan Priority: Critical Fix For: 4.2.0 When only one Host in the Cluster, I always success to create snapshot for the volume by manual or automation. But when have multiple Host in Cluster. The creating is failed with status "CreatedOnPrimary", sometimes it is successfull. The following is error log on Management when occur error, there are three error situations. ------> First Situation 2013-10-18 22:46:27,163 DEBUG [agent.transport.Request] (Job-Executor-122:job-120 = [ d07688b0- bd86-4259-bdc2-441c36c4727d ]) Seq 11-714015661: Received: { Ans: , MgmtId: 113353561884, via: 11, Ver: v1, Flags: 110, { CopyCmdAnswer } } 2013-10-18 22:46:27,175 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-122:job-120 = [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no snapshot with matching name '8a1d6db7-9ffe-43b5-a7df-df627329a168' at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-10-18 22:46:27,181 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-122:job-120 = [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Take snapshot: 12 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1040) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no snapshot with matching name '8a1d6db7-9ffe-43b5-a7df-df627329a168' at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) ... 16 more 2013-10-18 22:46:27,189 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-122:job-120 = [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Complete async job-120 = [ d07688b0-bd86-4259-bdc2- 441c36c4727d ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 12 ------> Second Situation 2013-10-18 23:30:23,010 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 8- 1133380250: Processing: { Ans: , MgmtId: 113353561884, via: 8, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup snapshot: /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh: line 121: 17900 Segmentation fault (core dumped) $qemu_img snapshot -d \"$snapshotname\" $diskFailed to delete snapshot 0c6ec28f-eaaf-487a-a96c-cd58e6771367 for path /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9- 924a3196cc08","wait":0}}] } 2013-10-18 23:30:23,010 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null) Seq 8- 1133380250: No more commands found 2013-10-18 23:30:23,010 DEBUG [agent.transport.Request] (Job-Executor-131:job-129 = [ 1bd4b8c8- 0680-470b-8577-baf7fbcf0fd2 ]) Seq 8-1133380250: Received: { Ans: , MgmtId: 113353561884, via: 8, Ver: v1, Flags: 110, { CopyCmdAnswer } } 2013-10-18 23:30:23,022 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-131:job-129 = [ 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot: /usr/share/cloudstack- common/scripts/storage/qcow2/managesnapshot.sh: line 121: 17900 Segmentation fault (core dumped) $qemu_img snapshot -d "$snapshotname" $diskFailed to delete snapshot 0c6ec28f-eaaf-487a- a96c-cd58e6771367 for path /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-10-18 23:30:23,028 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-131:job-129 = [ 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Take snapshot: 11 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1040) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot: /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh: line 121: 17900 Segmentation fault (core dumped) $qemu_img snapshot -d "$snapshotname" $diskFailed to delete snapshot 0c6ec28f-eaaf-487a-a96c-cd58e6771367 for path /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) ... 16 more 2013-10-18 23:30:23,036 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-131:job-129 = [ 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Complete async job-129 = [ 1bd4b8c8-0680-470b-8577- baf7fbcf0fd2 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 11 -----> Third Situation. 2013-10-19 10:39:03,220 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-43:job-206 = [ 661fcd14-8569-4e13-9e89-496adea6d054 ]) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: Failed to backup 8ef1d6d8-50af-4204-8f79- 8b8cfd907d50 for disk /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08 to /mnt/144a992e-7532- 316b-a15d-8e0ae22724c0/snapshots/2/11 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-10-19 10:39:03,227 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-43:job-206 = [ 661fcd14-8569-4e13-9e89-496adea6d054 ]) Take snapshot: 11 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1040) at com.cloud.utils.component.ComponentInstantiationPostProcessor $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot (VolumeServiceImpl.java:1307) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute (CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 8ef1d6d8-50af-4204- 8f79-8b8cfd907d50 for disk /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08 to /mnt/144a992e-7532 -316b-a15d-8e0ae22724c0/snapshots/2/11 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot (SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot (XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot (XenserverSnapshotStrategy.java:264) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot (SnapshotManagerImpl.java:1013) ... 16 more 2013-10-19 10:39:03,235 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-43:job-206 = [ 661fcd14-8569-4e13-9e89-496adea6d054 ]) Complete async job-206 = [ 661fcd14-8569-4e13-9e89- 496adea6d054 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 11 -- This message was sent by Atlassian JIRA (v6.1#6144)