Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id F21EC200BDB for ; Mon, 28 Nov 2016 00:12:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id F0B3B160B20; Sun, 27 Nov 2016 23:12:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7E7FF160B12 for ; Mon, 28 Nov 2016 00:11:59 +0100 (CET) Received: (qmail 58661 invoked by uid 500); 27 Nov 2016 23:11:58 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 58652 invoked by uid 500); 27 Nov 2016 23:11:58 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 58649 invoked by uid 99); 27 Nov 2016 23:11:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Nov 2016 23:11:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 79D422C03DF for ; Sun, 27 Nov 2016 23:11:58 +0000 (UTC) Date: Sun, 27 Nov 2016 23:11:58 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-9538) Deleting Snapshot From Primary Storage Fails on RBD Storage if you already delete vm's itself MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Sun, 27 Nov 2016 23:12:01 -0000 [ https://issues.apache.org/jira/browse/CLOUDSTACK-9538?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D15700472#comment-15700472 ]=20 ASF GitHub Bot commented on CLOUDSTACK-9538: -------------------------------------------- Github user blueorangutan commented on the issue: https://github.com/apache/cloudstack/pull/1710 =20 Trillian test result (tid-462) Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7 Total time taken: 26823 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download= /trillian/pr1710-t462-kvm-centos7.zip Test completed. 46 look ok, 2 have error(s) =20 =20 Test | Result | Time (s) | Test File --- | --- | --- | --- test_02_redundant_VPC_default_routes | `Failure` | 873.43 | test_vpc_re= dundant.py test_router_dhcp_opts | `Failure` | 20.89 | test_router_dhcphosts.py test_01_vpc_site2site_vpn | Success | 144.25 | test_vpc_vpn.py test_01_vpc_remote_access_vpn | Success | 65.83 | test_vpc_vpn.py test_01_redundant_vpc_site2site_vpn | Success | 229.84 | test_vpc_vpn.p= y test_02_VPC_default_routes | Success | 278.50 | test_vpc_router_nics.py test_01_VPC_nics_after_destroy | Success | 542.32 | test_vpc_router_nic= s.py test_05_rvpc_multi_tiers | Success | 499.58 | test_vpc_redundant.py test_04_rvpc_network_garbage_collector_nics | Success | 1400.05 | test_= vpc_redundant.py test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | S= uccess | 532.40 | test_vpc_redundant.py test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Success | 1273.= 02 | test_vpc_redundant.py test_09_delete_detached_volume | Success | 10.36 | test_volumes.py test_08_resize_volume | Success | 15.30 | test_volumes.py test_07_resize_fail | Success | 20.35 | test_volumes.py test_06_download_detached_volume | Success | 15.34 | test_volumes.py test_05_detach_volume | Success | 100.26 | test_volumes.py test_04_delete_attached_volume | Success | 10.16 | test_volumes.py test_03_download_attached_volume | Success | 15.26 | test_volumes.py test_02_attach_volume | Success | 73.79 | test_volumes.py test_01_create_volume | Success | 651.67 | test_volumes.py test_deploy_vm_multiple | Success | 282.83 | test_vm_life_cycle.py test_deploy_vm | Success | 0.02 | test_vm_life_cycle.py test_advZoneVirtualRouter | Success | 0.02 | test_vm_life_cycle.py test_10_attachAndDetach_iso | Success | 26.70 | test_vm_life_cycle.py test_09_expunge_vm | Success | 125.17 | test_vm_life_cycle.py test_08_migrate_vm | Success | 30.62 | test_vm_life_cycle.py test_07_restore_vm | Success | 0.09 | test_vm_life_cycle.py test_06_destroy_vm | Success | 125.68 | test_vm_life_cycle.py test_03_reboot_vm | Success | 125.69 | test_vm_life_cycle.py test_02_start_vm | Success | 10.15 | test_vm_life_cycle.py test_01_stop_vm | Success | 40.27 | test_vm_life_cycle.py test_CreateTemplateWithDuplicateName | Success | 110.75 | test_template= s.py test_08_list_system_templates | Success | 0.02 | test_templates.py test_07_list_public_templates | Success | 0.03 | test_templates.py test_05_template_permissions | Success | 0.04 | test_templates.py test_04_extract_template | Success | 5.22 | test_templates.py test_03_delete_template | Success | 5.08 | test_templates.py test_02_edit_template | Success | 90.16 | test_templates.py test_01_create_template | Success | 45.34 | test_templates.py test_10_destroy_cpvm | Success | 161.56 | test_ssvm.py test_09_destroy_ssvm | Success | 163.55 | test_ssvm.py test_08_reboot_cpvm | Success | 101.51 | test_ssvm.py test_07_reboot_ssvm | Success | 133.00 | test_ssvm.py test_06_stop_cpvm | Success | 106.78 | test_ssvm.py test_05_stop_ssvm | Success | 133.15 | test_ssvm.py test_04_cpvm_internals | Success | 1.20 | test_ssvm.py test_03_ssvm_internals | Success | 2.92 | test_ssvm.py test_02_list_cpvm_vm | Success | 0.09 | test_ssvm.py test_01_list_sec_storage_vm | Success | 0.10 | test_ssvm.py test_01_snapshot_root_disk | Success | 16.05 | test_snapshots.py test_04_change_offering_small | Success | 239.42 | test_service_offerin= gs.py test_03_delete_service_offering | Success | 0.03 | test_service_offerin= gs.py test_02_edit_service_offering | Success | 0.06 | test_service_offerings= .py test_01_create_service_offering | Success | 0.08 | test_service_offerin= gs.py test_02_sys_template_ready | Success | 0.11 | test_secondary_storage.py test_01_sys_vm_start | Success | 0.13 | test_secondary_storage.py test_09_reboot_router | Success | 40.27 | test_routers.py test_08_start_router | Success | 30.22 | test_routers.py test_07_stop_router | Success | 10.13 | test_routers.py test_06_router_advanced | Success | 0.04 | test_routers.py test_05_router_basic | Success | 0.03 | test_routers.py test_04_restart_network_wo_cleanup | Success | 5.67 | test_routers.py test_03_restart_network_cleanup | Success | 60.39 | test_routers.py test_02_router_internal_adv | Success | 0.85 | test_routers.py test_01_router_internal_basic | Success | 0.49 | test_routers.py test_router_dns_guestipquery | Success | 76.73 | test_router_dns.py test_router_dns_externalipquery | Success | 0.05 | test_router_dns.py test_router_dhcphosts | Success | 271.88 | test_router_dhcphosts.py test_01_updatevolumedetail | Success | 0.06 | test_resource_detail.py test_01_reset_vm_on_reboot | Success | 165.94 | test_reset_vm_on_reboot= .py test_createRegion | Success | 0.03 | test_regions.py test_create_pvlan_network | Success | 5.16 | test_pvlan.py test_dedicatePublicIpRange | Success | 0.29 | test_public_ip_range.py test_04_rvpc_privategw_static_routes | Success | 531.26 | test_privateg= w_acl.py test_03_vpc_privategw_restart_vpc_cleanup | Success | 469.64 | test_pri= vategw_acl.py test_02_vpc_privategw_static_routes | Success | 465.81 | test_privategw= _acl.py test_01_vpc_privategw_acl | Success | 97.45 | test_privategw_acl.py test_01_primary_storage_nfs | Success | 35.99 | test_primary_storage.py test_createPortablePublicIPRange | Success | 15.14 | test_portable_publ= icip.py test_createPortablePublicIPAcquire | Success | 15.35 | test_portable_pu= blicip.py test_isolate_network_password_server | Success | 90.81 | test_password_= server.py test_UpdateStorageOverProvisioningFactor | Success | 0.10 | test_over_p= rovisioning.py test_oobm_zchange_password | Success | 20.51 | test_outofbandmanagement= .py test_oobm_multiple_mgmt_server_ownership | Success | 16.51 | test_outof= bandmanagement.py test_oobm_issue_power_status | Success | 10.51 | test_outofbandmanageme= nt.py test_oobm_issue_power_soft | Success | 15.52 | test_outofbandmanagement= .py test_oobm_issue_power_reset | Success | 15.51 | test_outofbandmanagemen= t.py test_oobm_issue_power_on | Success | 16.52 | test_outofbandmanagement.p= y test_oobm_issue_power_off | Success | 15.51 | test_outofbandmanagement.= py test_oobm_issue_power_cycle | Success | 10.51 | test_outofbandmanagemen= t.py test_oobm_enabledisable_across_clusterzones | Success | 52.28 | test_ou= tofbandmanagement.py test_oobm_enable_feature_valid | Success | 5.16 | test_outofbandmanagem= ent.py test_oobm_enable_feature_invalid | Success | 0.10 | test_outofbandmanag= ement.py test_oobm_disable_feature_valid | Success | 5.16 | test_outofbandmanage= ment.py test_oobm_disable_feature_invalid | Success | 0.10 | test_outofbandmana= gement.py test_oobm_configure_invalid_driver | Success | 0.09 | test_outofbandman= agement.py test_oobm_configure_default_driver | Success | 0.08 | test_outofbandman= agement.py test_oobm_background_powerstate_sync | Success | 29.80 | test_outofband= management.py test_extendPhysicalNetworkVlan | Success | 15.28 | test_non_contigiousv= lan.py test_01_nic | Success | 524.62 | test_nic.py test_releaseIP | Success | 323.64 | test_network.py test_reboot_router | Success | 438.81 | test_network.py test_public_ip_user_account | Success | 10.19 | test_network.py test_public_ip_admin_account | Success | 40.22 | test_network.py test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Success | = 66.73 | test_network.py test_network_rules_acquired_public_ip_2_nat_rule | Success | 61.59 | te= st_network.py test_network_rules_acquired_public_ip_1_static_nat_rule | Success | 124= .21 | test_network.py test_delete_account | Success | 317.84 | test_network.py test_02_port_fwd_on_non_src_nat | Success | 55.48 | test_network.py test_01_port_fwd_on_src_nat | Success | 111.63 | test_network.py test_nic_secondaryip_add_remove | Success | 242.70 | test_multipleips_p= er_nic.py login_test_saml_user | Success | 21.34 | test_login.py test_assign_and_removal_lb | Success | 133.40 | test_loadbalance.py test_02_create_lb_rule_non_nat | Success | 187.38 | test_loadbalance.py test_01_create_lb_rule_src_nat | Success | 219.22 | test_loadbalance.py test_03_list_snapshots | Success | 0.05 | test_list_ids_parameter.py test_02_list_templates | Success | 0.03 | test_list_ids_parameter.py test_01_list_volumes | Success | 0.02 | test_list_ids_parameter.py test_07_list_default_iso | Success | 0.05 | test_iso.py test_05_iso_permissions | Success | 0.04 | test_iso.py test_04_extract_Iso | Success | 5.34 | test_iso.py test_03_delete_iso | Success | 95.17 | test_iso.py test_02_edit_iso | Success | 0.06 | test_iso.py test_01_create_iso | Success | 21.41 | test_iso.py test_04_rvpc_internallb_haproxy_stats_on_all_interfaces | Success | 293= .18 | test_internal_lb.py test_03_vpc_internallb_haproxy_stats_on_all_interfaces | Success | 273.= 13 | test_internal_lb.py test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 | Success | 504.68 = | test_internal_lb.py test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Success | 413.41 |= test_internal_lb.py test_dedicateGuestVlanRange | Success | 10.23 | test_guest_vlan_range.p= y test_UpdateConfigParamWithScope | Success | 0.10 | test_global_settings= .py test_rolepermission_lifecycle_update | Success | 6.38 | test_dynamicrol= es.py test_rolepermission_lifecycle_list | Success | 6.27 | test_dynamicroles= .py test_rolepermission_lifecycle_delete | Success | 6.17 | test_dynamicrol= es.py test_rolepermission_lifecycle_create | Success | 6.18 | test_dynamicrol= es.py test_rolepermission_lifecycle_concurrent_updates | Success | 6.32 | tes= t_dynamicroles.py test_role_lifecycle_update_role_inuse | Success | 6.22 | test_dynamicro= les.py test_role_lifecycle_update | Success | 11.26 | test_dynamicroles.py test_role_lifecycle_list | Success | 6.20 | test_dynamicroles.py test_role_lifecycle_delete | Success | 11.23 | test_dynamicroles.py test_role_lifecycle_create | Success | 6.24 | test_dynamicroles.py test_role_inuse_deletion | Success | 6.25 | test_dynamicroles.py test_role_account_acls_multiple_mgmt_servers | Success | 7.58 | test_dy= namicroles.py test_role_account_acls | Success | 7.79 | test_dynamicroles.py test_default_role_deletion | Success | 6.30 | test_dynamicroles.py test_04_create_fat_type_disk_offering | Success | 0.06 | test_disk_offe= rings.py test_03_delete_disk_offering | Success | 0.04 | test_disk_offerings.py test_02_edit_disk_offering | Success | 0.04 | test_disk_offerings.py test_02_create_sparse_type_disk_offering | Success | 0.06 | test_disk_o= fferings.py test_01_create_disk_offering | Success | 0.08 | test_disk_offerings.py test_deployvm_userdispersing | Success | 60.66 | test_deploy_vms_with_v= aried_deploymentplanners.py test_deployvm_userconcentrated | Success | 20.42 | test_deploy_vms_with= _varied_deploymentplanners.py test_deployvm_firstfit | Success | 50.59 | test_deploy_vms_with_varied_= deploymentplanners.py test_deployvm_userdata_post | Success | 10.27 | test_deploy_vm_with_use= rdata.py test_deployvm_userdata | Success | 95.87 | test_deploy_vm_with_userdata= .py test_02_deploy_vm_root_resize | Success | 6.26 | test_deploy_vm_root_re= size.py test_01_deploy_vm_root_resize | Success | 6.26 | test_deploy_vm_root_re= size.py test_00_deploy_vm_root_resize | Success | 288.28 | test_deploy_vm_root_= resize.py test_deploy_vm_from_iso | Success | 207.41 | test_deploy_vm_iso.py test_DeployVmAntiAffinityGroup | Success | 65.71 | test_affinity_groups= .py test_03_delete_vm_snapshots | Skipped | 0.00 | test_vm_snapshots.py test_02_revert_vm_snapshots | Skipped | 0.00 | test_vm_snapshots.py test_01_test_vm_volume_snapshot | Skipped | 0.00 | test_vm_snapshots.py test_01_create_vm_snapshots | Skipped | 0.00 | test_vm_snapshots.py test_06_copy_template | Skipped | 0.00 | test_templates.py test_static_role_account_acls | Skipped | 0.02 | test_staticroles.py test_01_scale_vm | Skipped | 0.00 | test_scale_vm.py test_01_primary_storage_iscsi | Skipped | 0.03 | test_primary_storage.p= y test_06_copy_iso | Skipped | 0.00 | test_iso.py test_deploy_vgpu_enabled_vm | Skipped | 0.01 | test_deploy_vgpu_enabled= _vm.py > Deleting Snapshot From Primary Storage Fails on RBD Storage if you alread= y delete vm's itself > -------------------------------------------------------------------------= -------------------- > > Key: CLOUDSTACK-9538 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-953= 8 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: KVM, Snapshot, Storage Controller > Affects Versions: 4.9.0 > Environment: Ubuntu 14.04 Management Server + Ubuntu 14.04 KVM > Reporter: =C3=96zhan R=C3=BCzgar Karaman > > Hi; > We plan to store vm snapshots as vm backups on secondary storage while we= still destroyed/expunged related vm. The idea is good there was a bug whic= h blocks this idea to work and it was fixed with CLOUDSTACK-9297 bug.=20 > Normally with 4.9 release we expected this idea to work on our on our 4.9= ACS environment but we noticed that because we are using rbd as primary st= orage we need to fix one minor bug for this idea to work. > The problem occurs because CLOUDSTACK-8302 bug fixed on 4.9 release and i= t block our idea to work. If you destroy a vm which is on RBD Storage as pr= imary storage it also deletes any related snapshots of that vm on Primary R= BD Storage. So after vm destroy no disk file or snapshot file over RBD Stor= age. This is good for cleanup purposes on primary storage but XenserverSnap= shotStrategy.deleteSnapshot class method did not expect this to happen. > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSn= apshot method receives exception. The code tries 10 times on KVM node to re= move RBD snapshot but because there is no snapshot on RBD side it get excep= tion after 10 retries, it also spends nearly 5 minutes to delete snapshots = and after that it ends with an error like "Failed to delete snapshot" error= . > I think we need to disable snapshot cleanup on primary storage only for R= BD type Primary Storage if its related vm was already been destroyed. (Beca= use vm destroy stage removed all snapshots related to vm on primary storage= so there is no need to take any action on primary storage.) > We make some tests below to make this issue clear for bug. > 1) We create a vm with 3 snapshots on ACS. > mysql> select * from snapshot_store_ref where snapshot_id in (93,94,95); > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -------+--------------+---------+---------------------+-----------+ > | id | store_id | snapshot_id | created | last_updated | job= _id | store_role | size | physical_size | parent_snapshot_id | insta= ll_path |= state | update_count | ref_cnt | updated | volume_id | > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -------+--------------+---------+---------------------+-----------+ > | 185 | 1 | 93 | 2016-10-12 10:13:44 | NULL | NUL= L | Primary | 28991029248 | 28991029248 | 0 | cst4/= bb9ca3c7-96d6-4465-85b5-cd01f4d635f2/54008bf3-43dd-469d-91a7-4acd146d7b84 |= Ready | 2 | 0 | 2016-10-12 10:13:45 | 4774 | > | 186 | 1 | 93 | 2016-10-12 10:13:45 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/54008bf3-43dd-469d-91a7-4acd146d7b84 |= Ready | 2 | 0 | 2016-10-12 10:15:04 | 4774 | > | 187 | 1 | 94 | 2016-10-12 10:15:38 | NULL | NUL= L | Primary | 28991029248 | 28991029248 | 0 | cst4/= bb9ca3c7-96d6-4465-85b5-cd01f4d635f2/45fc4f44-b377-49c0-9264-5d813fefe93f |= Ready | 2 | 0 | 2016-10-12 10:15:39 | 4774 | > | 188 | 1 | 94 | 2016-10-12 10:15:39 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/45fc4f44-b377-49c0-9264-5d813fefe93f |= Ready | 2 | 0 | 2016-10-12 10:16:52 | 4774 | > | 189 | 1 | 95 | 2016-10-12 10:17:08 | NULL | NUL= L | Primary | 28991029248 | 28991029248 | 0 | cst4/= bb9ca3c7-96d6-4465-85b5-cd01f4d635f2/1c1890bf-5a55-4060-bf3f-cf064629dcdf |= Ready | 2 | 0 | 2016-10-12 10:17:09 | 4774 | > | 190 | 1 | 95 | 2016-10-12 10:17:09 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf |= Ready | 2 | 0 | 2016-10-12 10:18:23 | 4774 | > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -------+--------------+---------+---------------------+-----------+ > 6 rows in set (0.00 sec) > mysql> select * from snapshots where name like '%TolgaC72%'; > +----+----------------+------------+-----------+-----------+-------------= -----+----------+------+------------------+--------------------------------= ------+---------------+------------------+-------------+-------------------= --+---------+----------------+----------+------------+--------------+------= -----------+---------+-------+----------+----------+ > | id | data_center_id | account_id | domain_id | volume_id | disk_offerin= g_id | status | path | name | uuid = | snapshot_type | type_description | size | created = | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hyper= visor_type | version | s3_id | min_iops | max_iops | > +----+----------------+------------+-----------+-----------+-------------= -----+----------+------+------------------+--------------------------------= ------+---------------+------------------+-------------+-------------------= --+---------+----------------+----------+------------+--------------+------= -----------+---------+-------+----------+----------+ > | 93 | 1 | 2 | 1 | 4774 | = 19 | BackedUp | NULL | S1TolgaC72Test41 | 0f84acd8-9a5c-4de5-a58a-b528b4e= 72839 | 0 | MANUAL | 28991029248 | 2016-10-12 10:13:4= 2 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > | 94 | 1 | 2 | 1 | 4774 | = 19 | BackedUp | NULL | S2TolgaC72Test41 | 51ef5246-27ae-435e-92ac-7428bab= 64cb8 | 0 | MANUAL | 28991029248 | 2016-10-12 10:15:3= 6 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > | 95 | 1 | 2 | 1 | 4774 | = 19 | BackedUp | NULL | S3TolgaC72Test41 | e5550dd6-9a43-4dd7-959a-7aabf26= f0593 | 0 | MANUAL | 28991029248 | 2016-10-12 10:17:0= 6 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > +----+----------------+------------+-----------+-----------+-------------= -----+----------+------+------------------+--------------------------------= ------+---------------+------------------+-------------+-------------------= --+---------+----------------+----------+------------+--------------+------= -----------+---------+-------+----------+----------+ > 3 rows in set (0.00 sec) > mysql> select uuid, name from volumes where name=3D'ROOT-4795'; > +--------------------------------------+-----------+ > | uuid | name | > +--------------------------------------+-----------+ > | bb9ca3c7-96d6-4465-85b5-cd01f4d635f2 | ROOT-4795 | > +--------------------------------------+-----------+ > 1 row in set (0.00 sec) > mysql> > =20 > #VM + 3 Snapshots=20 > #Ceph / Primary Storage > root@mont1:/var/log/ceph# rbd ls -l -p cst4|grep bb9ca3c7-96d6-4465-85b5-= cd01f4d635f2 > bb9ca3c7-96d6-4465-85b5-cd01f4d635f2 = 27648M cst4/fb80c005-51d2-4fc4-8a98-39073866e826@cloudstack-base-snap 2 > bb9ca3c7-96d6-4465-85b5-cd01f4d635f2@54008bf3-43dd-469d-91a7-4acd146d7b84= 27648M cst4/fb80c005-51d2-4fc4-8a98-39073866e826@cloudstack-base-snap 2 > bb9ca3c7-96d6-4465-85b5-cd01f4d635f2@45fc4f44-b377-49c0-9264-5d813fefe93f= 27648M cst4/fb80c005-51d2-4fc4-8a98-39073866e826@cloudstack-base-snap 2 > bb9ca3c7-96d6-4465-85b5-cd01f4d635f2@1c1890bf-5a55-4060-bf3f-cf064629dcdf= 27648M cst4/fb80c005-51d2-4fc4-8a98-39073866e826@cloudstack-base-snap 2 > root@mont1:/var/log/ceph# > #NFS / Secondary Side > root@cloudstackt4:/CST/S# ls -l snapshots/2/4774/54008bf3-43dd-469d-91a7-= 4acd146d7b84 snapshots/2/4774/45fc4f44-b377-49c0-9264-5d813fefe93f snapshot= s/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf > -rw-r--r-- 1 root root 28991029248 Oct 12 13:18 snapshots/2/4774/1c1890bf= -5a55-4060-bf3f-cf064629dcdf > -rw-r--r-- 1 root root 28991029248 Oct 12 13:16 snapshots/2/4774/45fc4f44= -b377-49c0-9264-5d813fefe93f > -rw-r--r-- 1 root root 28991029248 Oct 12 13:15 snapshots/2/4774/54008bf3= -43dd-469d-91a7-4acd146d7b84 > root@cloudstackt4:/CST/S# > 2) We destroy only vm not snapshots so 3 snapshots are active with Backed= Up Status. On vm destroy stage vm destroy method removes all vm's files and= its related snapshots on primary storage. > #VM Destroyed we have 3 Snapshots on BackedUp state > #Ceph Side / Primary Storage > root@mont1:/var/log/ceph# rbd ls -l -p cst4|grep bb9ca3c7-96d6-4465-85b5-= cd01f4d635f2 > root@mont1:/var/log/ceph# > #NFS / Secondary Side > root@cloudstackt4:/CST/S# ls -l snapshots/2/4774/54008bf3-43dd-469d-91a7-= 4acd146d7b84 snapshots/2/4774/45fc4f44-b377-49c0-9264-5d813fefe93f snapshot= s/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf > -rw-r--r-- 1 root root 28991029248 Oct 12 13:18 snapshots/2/4774/1c1890bf= -5a55-4060-bf3f-cf064629dcdf > -rw-r--r-- 1 root root 28991029248 Oct 12 13:16 snapshots/2/4774/45fc4f44= -b377-49c0-9264-5d813fefe93f > -rw-r--r-- 1 root root 28991029248 Oct 12 13:15 snapshots/2/4774/54008bf3= -43dd-469d-91a7-4acd146d7b84 > root@cloudstackt4:/CST/S# > 3) We destroy 1 Snapshot named S3TolgaC72Test41 > #Ceph Side / Primary Storage > root@mont1:/var/log/ceph# rbd ls -l -p cst4|grep bb9ca3c7-96d6-4465-85b5-= cd01f4d635f2 > root@mont1:/var/log/ceph# > #NFS / Secondary Side > root@cloudstackt4:/CST/S# ls -l snapshots/2/4774/54008bf3-43dd-469d-91a7-= 4acd146d7b84 snapshots/2/4774/45fc4f44-b377-49c0-9264-5d813fefe93f snapshot= s/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf > ls: cannot access snapshots/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf: = No such file or directory > -rw-r--r-- 1 root root 28991029248 Oct 12 13:16 snapshots/2/4774/45fc4f44= -b377-49c0-9264-5d813fefe93f > -rw-r--r-- 1 root root 28991029248 Oct 12 13:15 snapshots/2/4774/54008bf3= -43dd-469d-91a7-4acd146d7b84 > #SQL Side > mysql> select * from snapshots where name like '%TolgaC72%' and name !=3D= 'StolgaC72Revert'; > +----+----------------+------------+-----------+-----------+-------------= -----+-----------+------+------------------+-------------------------------= -------+---------------+------------------+-------------+------------------= ---+---------+----------------+----------+------------+--------------+-----= ------------+---------+-------+----------+----------+ > | id | data_center_id | account_id | domain_id | volume_id | disk_offerin= g_id | status | path | name | uuid = | snapshot_type | type_description | size | created = | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hype= rvisor_type | version | s3_id | min_iops | max_iops | > +----+----------------+------------+-----------+-----------+-------------= -----+-----------+------+------------------+-------------------------------= -------+---------------+------------------+-------------+------------------= ---+---------+----------------+----------+------------+--------------+-----= ------------+---------+-------+----------+----------+ > | 93 | 1 | 2 | 1 | 4774 | = 19 | BackedUp | NULL | S1TolgaC72Test41 | 0f84acd8-9a5c-4de5-a58a-b528b4= e72839 | 0 | MANUAL | 28991029248 | 2016-10-12 10:13:= 42 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > | 94 | 1 | 2 | 1 | 4774 | = 19 | BackedUp | NULL | S2TolgaC72Test41 | 51ef5246-27ae-435e-92ac-7428ba= b64cb8 | 0 | MANUAL | 28991029248 | 2016-10-12 10:15:= 36 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > | 95 | 1 | 2 | 1 | 4774 | = 19 | Destroyed | NULL | S3TolgaC72Test41 | e5550dd6-9a43-4dd7-959a-7aabf2= 6f0593 | 0 | MANUAL | 28991029248 | 2016-10-12 10:17:= 06 | NULL | NULL | NULL | NULL | NULL | KVM = | 2.2 | NULL | NULL | NULL | > +----+----------------+------------+-----------+-----------+-------------= -----+-----------+------+------------------+-------------------------------= -------+---------------+------------------+-------------+------------------= ---+---------+----------------+----------+------------+--------------+-----= ------------+---------+-------+----------+----------+ > 3 rows in set (0.00 sec) > mysql> select * from snapshot_store_ref where snapshot_id in (93,94,95); > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -----------+--------------+---------+---------------------+-----------+ > | id | store_id | snapshot_id | created | last_updated | job= _id | store_role | size | physical_size | parent_snapshot_id | insta= ll_path |= state | update_count | ref_cnt | updated | volume_id | > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -----------+--------------+---------+---------------------+-----------+ > | 185 | 1 | 93 | 2016-10-12 10:13:44 | NULL | NUL= L | Primary | 28991029248 | 28991029248 | 0 | cst4/= bb9ca3c7-96d6-4465-85b5-cd01f4d635f2/54008bf3-43dd-469d-91a7-4acd146d7b84 |= Ready | 2 | 0 | 2016-10-12 10:13:45 | 4774 | > | 186 | 1 | 93 | 2016-10-12 10:13:45 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/54008bf3-43dd-469d-91a7-4acd146d7b84 |= Ready | 2 | 0 | 2016-10-12 10:15:04 | 4774 | > | 187 | 1 | 94 | 2016-10-12 10:15:38 | NULL | NUL= L | Primary | 28991029248 | 28991029248 | 0 | cst4/= bb9ca3c7-96d6-4465-85b5-cd01f4d635f2/45fc4f44-b377-49c0-9264-5d813fefe93f |= Ready | 2 | 0 | 2016-10-12 10:15:39 | 4774 | > | 188 | 1 | 94 | 2016-10-12 10:15:39 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/45fc4f44-b377-49c0-9264-5d813fefe93f |= Ready | 2 | 0 | 2016-10-12 10:16:52 | 4774 | > | 190 | 1 | 95 | 2016-10-12 10:17:09 | NULL | NUL= L | Image | 28991029248 | 28991029248 | 0 | snaps= hots/2/4774/1c1890bf-5a55-4060-bf3f-cf064629dcdf |= Destroyed | 4 | 0 | 2016-10-12 10:48:31 | 4774 | > +-----+----------+-------------+---------------------+--------------+----= ----+------------+-------------+---------------+--------------------+------= --------------------------------------------------------------------------+= -----------+--------------+---------+---------------------+-----------+ > 5 rows in set (0.00 sec) > mysql> > ERROR on Cloudstack Management Server: Failed to delete snapshot > 2016-10-12 13:54:02,911 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-70:ctx-06858bab job-34505 ctx-a6bc52fb) (logid:a2b92099) Faile= d to delete snapshot: > com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapsho= t > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.del= eteSnapshot(SnapshotServiceImpl.java:396) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshot(XenserverSnapshotStrategy.java:272) > at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(= SnapshotManagerImpl.java:454) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at org.apache.cloudstack.network.contrail.management.EventUtils$E= ventInterceptor.invoke(EventUtils.java:106) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:161) > at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInter= ceptor.java:51) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:161) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at com.sun.proxy.$Proxy192.deleteSnapshot(Unknown Source) > at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshot= Cmd.execute(DeleteSnapshotCmd.java:103) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatch= er.java:108) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.runInContext(AsyncJobManagerImpl.java:554) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.run(AsyncJobManagerImpl.java:502) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-10-12 13:54:02,918 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-70:ctx-06858bab job-34505) (logid:a2b92099) Complete async job-345= 05, jobStatus: FAILED, resultCode: 530, > result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuid= List":[],"errorcode":530,"errortext":"Failed to delete snapshot"} > ERROR on KVM Agent > 2016-10-12 13:49:38,227 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentR= equest-Handler-3:null) Could not find volume bb9ca3c7-96d6-4465-85b5-cd01f4= d635f2: Storage volume not found: no storage vol with matching name 'bb9ca3= c7-96d6-4465-85b5-cd01f4d635f2' > 2016-10-12 13:49:38,227 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentR= equest-Handler-3:null) Refreshing storage pool 01b375db-d3f5-33c1-9389-8bf2= 26c887e8 > 2016-10-12 13:49:41,253 DEBUG [kvm.storage.KVMStoragePoolManager] (agentR= equest-Handler-3:null) Failed to find volume:bb9ca3c7-96d6-4465-85b5-cd01f4= d635f2 due tocom.cloud.utils.exception.CloudRuntimeException: Could not fin= d volume bb9ca3c7-96d6-4465-85b5-cd01f4d635f2: Storage volume not found: no= storage vol with matching name 'bb9ca3c7-96d6-4465-85b5-cd01f4d635f2', ret= ry:2 > 2016-10-12 13:51:53,819 DEBUG [kvm.storage.KVMStoragePoolManager] (agentR= equest-Handler-3:null) Failed to find volume:bb9ca3c7-96d6-4465-85b5-cd01f4= d635f2 due tocom.cloud.utils.exception.CloudRuntimeException: Could not fin= d volume bb9ca3c7-96d6-4465-85b5-cd01f4d635f2: Storage volume not found: no= storage vol with matching name 'bb9ca3c7-96d6-4465-85b5-cd01f4d635f2', ret= ry:6 > 2016-10-12 13:52:26,858 DEBUG [kvm.storage.KVMStoragePoolManager] (agentR= equest-Handler-3:null) Failed to find volume:bb9ca3c7-96d6-4465-85b5-cd01f4= d635f2 due tocom.cloud.utils.exception.CloudRuntimeException: Could not fin= d volume bb9ca3c7-96d6-4465-85b5-cd01f4d635f2: Storage volume not found: no= storage vol with matching name 'bb9ca3c7-96d6-4465-85b5-cd01f4d635f2', ret= ry:7 > 2016-10-12 13:53:33,390 DEBUG [kvm.storage.KVMStoragePoolManager] (agentR= equest-Handler-3:null) Failed to find volume:bb9ca3c7-96d6-4465-85b5-cd01f4= d635f2 due tocom.cloud.utils.exception.CloudRuntimeException: Could not fin= d volume bb9ca3c7-96d6-4465-85b5-cd01f4d635f2: Storage volume not found: no= storage vol with matching name 'bb9ca3c7-96d6-4465-85b5-cd01f4d635f2', ret= ry:9 > 2016-10-12 13:54:03,390 ERROR [kvm.storage.KVMStorageProcessor] (agentReq= uest-Handler-3:null) Failed to remove snapshot , with exception: com.cloud.= utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRunti= meException: Could not find volume bb9ca3c7-96d6-4465-85b5-cd01f4d635f2: St= orage volume not found: no storage vol with matching name 'bb9ca3c7-96d6-44= 65-85b5-cd01f4d635f2' > Thanks > =C3=96zhan -- This message was sent by Atlassian JIRA (v6.3.4#6332)