Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
From: Pavan Bandarupally <pavan.bandarupally@accelerite.com>
To: =?iso-8859-1?Q?S=2E_Br=FCseke_-_proIO_GmbH?= <s.brueseke@proio.com>,
	"users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: RE: snapshot cleanup on hypervisor primary storage
Thread-Topic: snapshot cleanup on hypervisor primary storage
Thread-Index: AdGPD+huw0iASAcJTTaRBgeVuT0/XAACs92QAAGojYAAAddVwAABKUMQAACt4CA=
Date: Tue, 5 Apr 2016 12:03:00 +0000
Message-ID: 
 <MA1PR01MB026211A41283D435ED1B063CF39E0@MA1PR01MB0262.INDPRD01.PROD.OUTLOOK.COM>
References: <c36e2e86-c27a-41f9-b5d1-bd346c669244@proio.com>
	<MA1PR01MB026279FF6011FD09BD20FABFF39E0@MA1PR01MB0262.INDPRD01.PROD.OUTLOOK.COM>
	<fe1ed86e-8a0d-49e6-8a68-b2f14fe1f04b@proio.com>
	<MA1PR01MB026277EAB2B1C9852A0477E0F39E0@MA1PR01MB0262.INDPRD01.PROD.OUTLOOK.COM>
 <95dc4069-e780-4df9-bcbd-a805add14dd4@proio.com>
In-Reply-To: <95dc4069-e780-4df9-bcbd-a805add14dd4@proio.com>
Accept-Language: en-US
Content-Language: en-US
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
content-transfer-encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Apr 2016 12:03:00.8512
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 1f4beacd-b7aa-49b2-aaa1-b8525cb257e0
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MA1PR01MB0261
X-OriginatorOrg: accelerite.com
X-Brightmail-Tracker: 
 H4sIAAAAAAAAA+NgFtrOJsWRmVeSWpSXmKPExsXCpfPHUbdiJXO4wdtnwhZ/Pk1mt3gx8Sub
	A5PHvuZOJo+Xbw8wBjBFNTDaJObl5ZcklqQqpKQWJ9sqOSYnp+akFmUCBXxS0xNzFFwyi5Nz
	EjNzU4uUFDJTbJVMlBQKchKTU3NT80pslRILClLzUpTsuBQwgA1QWWaeQmpecn5KZl66rZJn
	sL+uhYWppa6hkp2asqGxNReQTHjPmnH7V1DB3sCKJxM/MDcwTnDqYuTkkBAwkZj6ZBEThC0m
	ceHeerYuRi4OIYEtTBJX5hxmh3AOMUosX3maEcRhEZjNLLFu1m5GiMxVRonXnZuZIJyjjBJv
	V/9kAxnGJmAjcfveLFaQhIhAG6PEhnudLCAJYQFriRubDgN1cAAlbCTu/YoHCYsI+Em8fvyN
	GcRmEVCRuLrnBdgcXoEYiT+Nu6COOsEk8fvzJnaQBKeArUTrue1gRYxAl38/tQbsC2YBcYlb
	T+ZDfSQgsWTPeWYIW1Ti5eN/YAcxCvQzSqx/dIYFIqEscerZR0YI21eic8UTsN8kBPYwSVw4
	9BqqyE3i5NE1UJOyJaZtOgPVoCXRcWQW1LbljBJLNglA2DISBxashBr0lkXi0ePdjBDvS0k0
	7lkJDQoZiRd39rJOYNScheRyCFtP4sbUKWwQtrbEsoWvmWeBg0NQ4uTMJywLGFlWMcp4eOkG
	+4YE6PqHhugVpBYVZxaXAFOLXnK+XmbeJkZQ6tTlENzB2LNf/xCjAAejEg/vzfnM4UKsiWXF
	lbnAOOVgVhLhTV0BFOJNSaysSi3Kjy8qzUktPsToA4yEicxSosn5eSAj4w2NLS0tDczNjI3N
	DA1xCCuJ836U+hgmJJAOTObZqalAF8GMY+LglGpgXGD6hGHpxIJttycIWt1TK5LSWH/0i/I1
	tjiv09v5Xq61u+VS0qdV+WNnuHeAgiXbk3OyQVWrNqcsuvMmdj+X5vX3l2o4hQ40KvdavfjD
	9M5snvDHnRKcHp2nuy0CuJmPpW7Wt4g2EGi6tlowKUPs5c9nVXr/83hXPzXMZVSMjnZb3XdH
	956yEktxRqKhFnNRcSIA6e1Xl8oDAAA=

Hi Swen,

Cloud Stack just hands over the snapshot operation to XenServer via XenServe=
r commands. As mentioned earlier if the volume (VDI/VBD) is deleted all the=
 associated snapshots will be cleaned up by Xenserver. 

My assumption here is that the volume that you have deleted from cloud stack=
 is not deleted in the XenServer and hence the snapshots are not getting del=
eted ? Can you please check on XenServer if the VDI chain (for the deleted v=
olume) is still present in the SR ?

I am not exactly aware of any changes in functionality from 4.5.1 but you ca=
n try one thing to isolate if the issue is with CS version or XenServer. Can=
 you please create a standalone VM on Xenserver and take a snapshot of the V=
DI ? Delete the VDI and check if the snapshots are getting deleted ?

Regards,
Pavan
-----Original Message-----
From: S. Br=FCseke - proIO GmbH [mailto:s.brueseke@proio.com] 
Sent: Tuesday, April 05, 2016 4:54 PM
To: Pavan Bandarupally; users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage

Hi Pavan,

XenServer is not deleting the snapshots even after do a rescan:

Apr  5 13:20:59 cp-test-xs-2 SMGC: [24074] No work, exiting Apr  5 13:20:59=
 cp-test-xs-2 SMGC: [24074] In cleanup

I am really not sure why he should do it in first place. Can you please tell=
 me more about it? Please have in mind that we are using CS 4.5.1 and not th=
e newest version.

Mit freundlichen Gr=FC=DFen / With kind regards,

Swen


-----Urspr=FCngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 13:14
An: users@cloudstack.apache.org; S. Br=FCseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage

Hi Swen,

Yes , if you have deleted the volume, the snapshots of that volume should be=
 cleaned from primary storage by Xenserver. 

Can you please run sr scan and check if they are getting cleaned up (running=
 sr scan will manually trigger the GC on Xenserver which should clean up the=
se if GC is not run for some reason) ?

Regards,
Pavan
-----Original Message-----
From: S. Br=FCseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 4:05 PM
To: users@cloudstack.apache.org
Subject: AW: snapshot cleanup on hypervisor primary storage

Hi Pavan,

snapshots are working fine. Are you sure that snapshots on primary storage s=
hould be deleted?

I did some testing and observed the following:

First volume-snapshot of an instance
1. you create a volume-snapshot in CS UI 2. XenServer is taking a snapshot o=
f the vm 3. XenServer is mounting secondary storage 4. XenServer is copying=
 snapshot to secondary storage 5. XenServer is unmounting secondary storage

Second (or more) volume-snapshot of an instance 1. you create a volume-snaps=
hot in CS UI 2. XenServer is taking a snapshot of the vm and uses the first=
 snapshot as parent (you are unable to see this in XenCenter because the par=
ent snapshot will be hidden) 3. XenServer is mounting secondary storage 4. X=
enServer is copying only delta of snapshot to secondary storage into the sam=
e directory as the first snapshot 5. XenServer is unmounting secondary stora=
ge


If you delete the a snapshot via XenCenter all following volume-snapshots ar=
e going to fail because of missing parent!

If you migrate vm to another XenServer host and create a volume-snapshot it=
 will work. Reason for that is that CS is starting from the beginning here a=
nd handles this as a first snapshot. CS also uses a new folder on secondary=
 storage for this volume-snapshot.

The last snapshot on XenServer will always be there, because CS needs it as=
 parent for the next one and XenServer is creating a vhd chain.

The questions here are:
1. When I delete an instance is the snapshot on XenServer still needed?
I think now, it can be deleted even the volume-snapshot is still on secondar=
y storage.

2. When I delete all volume-snapshots in CS UI of the snapshot-chain will CS=
 delete the snapshots on XenServer?
As far as I can see, no.

3. Who is cleaning up the snapshots on XenServer's primary storage when you=
 do a storage migration (normal and live) to another primary storage?
Right now, nobody is doing this and if you are using storage migration a lot=
 (if you are using local storage you will do it a lot) than you end up with=
 GBs of unwanted data on your primary storages.

Mit freundlichen Gr=FC=DFen / With kind regards,

Swen


-----Urspr=FCngliche Nachricht-----
Von: Pavan Bandarupally [mailto:pavan.bandarupally@accelerite.com]
Gesendet: Dienstag, 5. April 2016 11:17
An: users@cloudstack.apache.org; S. Br=FCseke - proIO GmbH
Betreff: RE: snapshot cleanup on hypervisor primary storage

Hi Swen,

I don't think this is expected. The snapshots should get cleaned from Primar=
y Storage by XenServer. Can you please check if the snapshots are usable or=
 corrupted ? 

Regarding deletion of snapshots on expunging the VM is not expected, because=
 we do keep the snapshots (in secondary store) for further usage as template=
s / volumes can be created from the snapshots irrespective of whether the VM=
 from whose disk the snapshot was created exists or not.

Regards,
Pavan

-----Original Message-----
From: S. Br=FCseke - proIO GmbH [mailto:s.brueseke@proio.com]
Sent: Tuesday, April 05, 2016 1:21 PM
To: users@cloudstack.apache.org
Subject: snapshot cleanup on hypervisor primary storage

Hey guys,

we are using CS 4.5 with XenServer 6.5 SP1 and observed a behavior with volu=
me-snapshots that will fill up your primary storage over time:

Workflow:
1. create an instance
2. create a volume-snapshot of the ROOT-disk of that instance 3. delete inst=
ance and expunge it 4. check primary storage of XenServer. The latest snapsh=
ot of each volume-snapshot will stay on primary storage and is not being del=
eted even after waiting for storage.cleanup.interval

Can somebody reproduce this?

As far as I understand the workflow of volume-snapshots CS will XenServer as=
k to do a snapshot of the vm and then copy this snapshot to secondary storag=
e. But why CS is not deleting the snapshot on primary storage after a succes=
s copy to secondary storage? Is this a "broken" workflow or is there a reaso=
n for that?

Is this the same behavior in newer releases of CS?

Mit freundlichen Gr=FC=DFen / With kind regards,

Swen


- proIO GmbH -
Gesch=E4ftsf=FChrer: Swen Br=FCseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informat=
ionen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FCmlich erh=
alten haben, informieren Sie bitte sofort den Absender und vernichten Sie di=
ese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nich=
t gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error=
) please notify the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this=
 e-mail is strictly forbidden. 


DISCLAIMER
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This e-mail may contain privileged and confidential information which is the=
 property of Accelerite, a Persistent Systems business. It is intended only=
 for the use of the individual or entity to which it is addressed. If you ar=
e not the intended recipient, you are not authorized to read, retain, copy,=
 print, distribute or use this message. If you have received this communicat=
ion in error, please notify the sender and delete all copies of this message=
. Accelerite, a Persistent Systems business does not accept any liability fo=
r virus infected mails.


- proIO GmbH -
Gesch=E4ftsf=FChrer: Swen Br=FCseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informat=
ionen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FCmlich erh=
alten haben, informieren Sie bitte sofort den Absender und vernichten Sie di=
ese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nich=
t gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error=
) please notify the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this=
 e-mail is strictly forbidden. 


DISCLAIMER
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This e-mail may contain privileged and confidential information which is the=
 property of Accelerite, a Persistent Systems business. It is intended only=
 for the use of the individual or entity to which it is addressed. If you ar=
e not the intended recipient, you are not authorized to read, retain, copy,=
 print, distribute or use this message. If you have received this communicat=
ion in error, please notify the sender and delete all copies of this message=
. Accelerite, a Persistent Systems business does not accept any liability fo=
r virus infected mails.


- proIO GmbH -
Gesch=E4ftsf=FChrer: Swen Br=FCseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informat=
ionen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FCmlich erh=
alten haben, informieren Sie bitte sofort den Absender und vernichten Sie di=
ese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nich=
t gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error=
) please notify the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this=
 e-mail is strictly forbidden. 


DISCLAIMER
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This e-mail may contain privileged and confidential information which is the=
 property of Accelerite, a Persistent Systems business. It is intended only=
 for the use of the individual or entity to which it is addressed. If you ar=
e not the intended recipient, you are not authorized to read, retain, copy,=
 print, distribute or use this message. If you have received this communicat=
ion in error, please notify the sender and delete all copies of this message=
. Accelerite, a Persistent Systems business does not accept any liability fo=
r virus infected mails.