Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 647D9CDD7 for ; Thu, 11 Dec 2014 16:54:14 +0000 (UTC) Received: (qmail 28708 invoked by uid 500); 11 Dec 2014 16:54:14 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 28676 invoked by uid 500); 11 Dec 2014 16:54:14 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 28666 invoked by uid 500); 11 Dec 2014 16:54:14 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 28663 invoked by uid 99); 11 Dec 2014 16:54:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Dec 2014 16:54:14 +0000 Date: Thu, 11 Dec 2014 16:54:14 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-5446) KVM-Secondary Store down-Even after secondary store is brought back up after being down for few hours,snapshot jobs do not get triggered with reason "there is other active snapshot tasks on the instance to which the volume is attached" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242774#comment-14242774 ] ASF subversion and git services commented on CLOUDSTACK-5446: ------------------------------------------------------------- Commit 40bc5df0b1f82e5282521a869648b17c4af51558 in cloudstack's branch refs/heads/4.4 from Edison Su [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=40bc5df ] CLOUDSTACK-5446: delete all the leftover snapshots on primary storage in case of snapshot errors, after a new backup snapshot is finished (cherry picked from commit 2667855ccb932f9a03ddf6639f6411c73fea1b2c) Signed-off-by: Rohit Yadav > KVM-Secondary Store down-Even after secondary store is brought back up after being down for few hours,snapshot jobs do not get triggered with reason "there is other active snapshot tasks on the instance to which the volume is attached" > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-5446 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5446 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Management Server > Affects Versions: 4.3.0 > Environment: Build from 4.3 > Reporter: Sangeetha Hariharan > Assignee: edison su > Fix For: 4.4.0, 4.5.0 > > Attachments: agentdown.rar, ssdown.rar > > > KVM - Secondary Store down - Even after secondary store is brought back up after being down for few hours , snapshot jobs do not get triggered with reason "here is other active snapshot tasks on the instance to which the volume is attached, please try again later" > Set up: > Advanced Zone set up with 2 KVM (RHEL 6.3) hosts. > Steps to reproduce the problem: > 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms. > 2. Start concurrent snapshots for ROOT volumes of all the Vms. > 3. Shutdown the Secondary storage server when the snapshots are in the progress. > 4. Bring the Secondary storage server up after ~ 12 hours. > When the secondary storage was down: > The snapshot jobs that were in progress timed out after 6 hours. > Even after this , I do not see the hourly snapshots being scheduled due to the following reason: > > 2013-12-10 13:07:49,285 WARN [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-cf0775f7) Scheduling snapshot failed due to com.cloud.utils.exception.CloudRuntimeException: There is other active snapshot tasks on the instance to which the volume is attached, please try again later > Even after the secondary storage was brought up , there is no hourly snapshots being scheduled due to the same reason - "com.cloud.utils.exception.CloudRuntimeException: There is other active snapshot tasks on the instance to which the volume is attached, please try again later" > mysql> select * FROM snapshots; > +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+ > | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id | status | path | name | uuid | snapshot_type | type_description | size | created | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hypervisor_type | version | s3_id | > +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+ > | 1 | 1 | 6 | 1 | 45 | 18 | BackedUp | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131209234410 | ee2c46b7-7623-439a-9a30-63eec1d95c56 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 2 | 1 | 6 | 1 | 43 | 18 | BackedUp | NULL | TestVM-1_ROOT-43_20131209234410 | 62c01389-49df-475b-b225-617d3903d25a | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 3 | 1 | 6 | 1 | 54 | 19 | BackedUp | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131209234410 | 8c7621f0-5d94-4f22-b3f8-7954c810e1be | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 4 | 1 | 6 | 1 | 53 | 19 | BackedUp | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131209234410 | 30c72de6-2a3e-4bf2-88da-b2a3b6ae9c66 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 5 | 1 | 6 | 1 | 52 | 19 | BackedUp | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131209234411 | e8e9f99a-5a8b-4cfd-b743-327925955980 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 6 | 1 | 6 | 1 | 51 | 19 | BackedUp | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131209234411 | 07e4edc8-b289-4f94-8696-ccf9db60e3ab | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 7 | 1 | 6 | 1 | 50 | 19 | BackedUp | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131209234411 | b285aa1a-6650-4b90-9c2c-e8bf0c507bfb | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 8 | 1 | 6 | 1 | 49 | 18 | BackedUp | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131209234412 | 31e37c8d-aab2-4ee6-9c43-c23501b08b02 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 9 | 1 | 6 | 1 | 48 | 18 | BackedUp | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131209234412 | 60b6023e-138f-42c2-9867-f559e1d95352 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 10 | 1 | 6 | 1 | 47 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131209234412 | fa37ebe5-f36c-4f19-b99f-249ab95c4e61 | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 11 | 1 | 6 | 1 | 46 | 18 | BackedUp | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131209234412 | bd571765-b28e-433e-b480-ccdc3bc0450b | 3 | HOURLY | 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 12 | 1 | 6 | 1 | 45 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210004410 | 4c6f50c2-661e-4bad-9f07-357b2ca4e7fc | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 13 | 1 | 6 | 1 | 43 | 18 | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210004410 | bd3744ae-182f-4a0b-9685-5bef0c747365 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 14 | 1 | 6 | 1 | 54 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210004410 | 14f53a2f-bb4b-40e5-aa90-e7b4be4a1076 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 15 | 1 | 6 | 1 | 53 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210004410 | 88dd82f5-ade5-4dd5-a147-e0a35a15be85 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 16 | 1 | 6 | 1 | 52 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210004410 | 1830af7d-97e5-488c-bc37-39c11961e509 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 17 | 1 | 6 | 1 | 51 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210004410 | 3d452970-f042-4af6-a8cc-3003482da9a8 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 18 | 1 | 6 | 1 | 50 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210004410 | 0cb1b176-bcfe-4bc2-9b5e-07ec8e9e97f5 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 19 | 1 | 6 | 1 | 49 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210004411 | 87337d8b-e698-47b1-961a-32479d44a6e5 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 20 | 1 | 6 | 1 | 48 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210004411 | c7b321bd-31b4-4c9c-be8f-c9231cb08530 | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 21 | 1 | 6 | 1 | 47 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131210004411 | ea1a71b0-9b52-42e5-a252-7a3e8938bc0f | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 22 | 1 | 6 | 1 | 46 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210004411 | f1fbc899-5f55-472c-b798-639350b2facd | 3 | HOURLY | 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 33 | 1 | 6 | 1 | 45 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210022749 | 396c80a8-299c-42b2-8c3d-32da5af9536c | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 34 | 1 | 6 | 1 | 43 | 18 | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210022749 | 9444a355-08a6-4c7d-98f2-74729a0b28b5 | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 35 | 1 | 6 | 1 | 54 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210022749 | 09cbfc03-7f9f-470f-bc34-e2bbad72b9ba | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 36 | 1 | 6 | 1 | 53 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210022749 | 2e7259f0-82d2-4910-9047-617d54bda035 | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 37 | 1 | 6 | 1 | 52 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210022749 | 420230ed-a037-4ed3-a019-031d24fd0bfb | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 38 | 1 | 6 | 1 | 51 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210022750 | 6bf87e88-3a14-467f-ac5b-ab620fd03206 | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 39 | 1 | 6 | 1 | 50 | 19 | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210022750 | 93021350-9df1-4234-9df2-869d9e9a912f | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 40 | 1 | 6 | 1 | 49 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210022750 | 4699ebdf-3093-4f2f-86b7-ae5c9195cc6a | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 41 | 1 | 6 | 1 | 48 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210022750 | 8d663979-31a2-48ad-884b-f1ef85a1f4d5 | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > | 42 | 1 | 6 | 1 | 46 | 18 | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210022751 | f80c2994-b612-4234-add4-27d323cf3dd3 | 3 | HOURLY | 21474836480 | 2013-12-10 02:27:51 | NULL | NULL | NULL | NULL | NULL | KVM | 2.2 | NULL | > +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+ > 32 rows in set (0.00 sec) > mysql> -- This message was sent by Atlassian JIRA (v6.3.4#6332)