cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [Propose][New Feature] Storage Snapshots
Date Fri, 05 Feb 2016 14:44:54 GMT
Hey Pierre-Luc,

For me, it's not a matter of how much work has already been done, but
rather maintaining backward compatibility.

Today, if using managed storage, if you elect to create a volume snapshot
(and the storage driver supports this), it is only taken on the backend
system (in my case, the SAN)...it's not backed up to NFS.

We just need to maintain that behavior (although we can extend it with a
flag or something that tells us to also back the volume snapshot up to
NFS...that would be perfectly fine).

What do you think about that?

In summary:

* Non-managed storage (traditional storage where an admin pre-creates an SR
(or datastore), hands it to CS as primary storage, and then runs multiple
virtual disks on it) implements volume snapshots as backups to NFS.

* Managed storage (1:1 mapping between an SR (or datastore) and a virtual
disk) implements volume snapshots as snapshots on the backend system (a
SAN, for me). Volume snapshots for managed storage currently is only
implemented for XenServer (first in 4.6).

Thanks,
Mike

On Fri, Feb 5, 2016 at 5:56 AM, Pierre-Luc Dion <pdion@cloudops.com> wrote:

> Hi Mike,
>
> The idea of introducing a new API: StorageSnapshot for managed storage is
> because the VolumeSnapshot default, or expected, behavior is to archive
> snapshots into the Secondary Storage. So a StorageSnapshot API would be for
> snapshot that remain on the managed storage appliance.
>
> Quickly looking at the API doc and I don't see a strong requirement for
> volume snapshots to be moved into secondary storage. So, maybe
> StorageSnapshot API is not useful, but both use cases are required. A
> snapshot that remain on the managed storage, and another type of snapshot
> that end up into the secondary storage. Since you've done a lot of work,
> might easier to just add a parameter to the current snapshot API that would
> trigger an extraction of the storage snapshot into the secondary storage?
>
> PL
>
>
>
> On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > I think that all sounds reasonable then - thanks!
> >
> > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <syed1.mushtaq@gmail.com>
> > wrote:
> >
> >> You are correct Mike in terms of the requirements. One of our earlier
> >> iterations on this was to have an argument to the create snapshot API
> which
> >> decides whether to backup the volume to sec storage but we realized it
> >> would make management of snapshots quite messy so we proposed a new api
> >> instead.
> >>
> >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Just to make sure I understand all the requirements here:
> >>>
> >>> 1) This relates only to managed storage (1:1 mapping between a virtual
> >>> disk and a backend SAN volume).
> >>>
> >>> 2) We want to take the current (introduced in 4.6) functionality, which
> >>> creates a snapshot on the SAN, and extend it via a config option (or
> >>> something) to not only take the SAN snapshot, but to copy the
> underlying
> >>> VHD (XenServer only) to NFS.
> >>>
> >>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> >>> optional.
> >>>
> >>> 4) Templates can be created from the snapshot that's on the SAN
> (already
> >>> works).
> >>>
> >>> 5) CloudStack volumes can be created from the snapshot that's on the
> SAN
> >>> (already works as long as the new CloudStack volume ends up on the same
> >>> primary storage).
> >>>
> >>> Would we have a need for a storage snapshot API then or would that just
> >>> be the standard volume snapshot without the backup to NFS?
> >>>
> >>> Thanks!
> >>> Mike
> >>>
> >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <syed1.mushtaq@gmail.com>
> >>> wrote:
> >>>
> >>>> Is it possible to have both functionalities (snapshot on SAN & Sec
> >>>> Storage) coexist? Because Ideally, we would like to have both.
> >>>> For example, some of our customers want to implement their own backup
> >>>> strategies and do encryption to their backups which is a perfect
> >>>> use case for Storage Snapshot while our other customers will still
> keep
> >>>> using the standard volume snapshot.
> >>>>
> >>>> To keep things backward compatible, we can add a setting which says
to
> >>>> not upload on secondary storage, because, after all, you would take
a
> >>>> SAN snapshot first when doing a Volume Snapshot. You could stop the
> >>>> process there and not do the upload.
> >>>>
> >>>> What do you think about this approach?
> >>>>
> >>>> Thanks,
> >>>> -Syed
> >>>>
> >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com> wrote:
> >>>>
> >>>>> So, this is just me thinking out load here, but if a given CloudStack
> >>>>> cloud doesn't actually need to provide both the ability to take
a SAN
> >>>>> snapshot and export it to NFS (if just taking a SAN snapshot is
OK),
> then
> >>>>> we might be able to get away with no new API calls and simply
> implement a
> >>>>> new custom snapshot strategy and data motion strategy to handle
the
> case
> >>>>> where the CloudStack cloud does want both a SAN snapshot and
> >>>>> exported-to-NFS backup.
> >>>>>
> >>>>> In other words, the "default" behavior would be to use the snapshot
> >>>>> strategy and data motion strategy that we already have (the one
that
> only
> >>>>> takes a SAN snapshot).
> >>>>>
> >>>>> If your CloudStack cloud, however, wants to take a SAN snapshot
and
> >>>>> have the data exported to NFS, then we could have you manipulate
a
> Swing
> >>>>> config file to make use of a new snapshot strategy and data motion
> strategy
> >>>>> that performs both of these activities.
> >>>>>
> >>>>> This way, the old behavior is still the default for users, but
> >>>>> CloudStack admins can change this behavior via configuration.
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> >>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>
> >>>>>> Right...I think we will need to come up with a viable upgrade
path
> or
> >>>>>> some reasonable way for them to move from the old way to the
new
> way (and
> >>>>>> some obvious way that they will know they need to do this).
> >>>>>>
> >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> >>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>
> >>>>>>> I'm not really sure about the upgrade path however, customers
who
> >>>>>>> are using 4.6 and are on a managed storage would no longer
have
> the same
> >>>>>>> functionality with Volume Snapshots.
> >>>>>>>
> >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> >>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> So if I understand correctly, currently taking a Volume
Snapshots
> >>>>>>>> of a volume on a managed storage keeps it on the storage
array.
> As a part
> >>>>>>>> of this feature, we can make sure that Volume Snapshots
on
> managed storage
> >>>>>>>> are uploaded to the secondary storage. This would make
the Volume
> Snapshot
> >>>>>>>> feature behave the same regardless of the storage (managed
or
> non-managed)
> >>>>>>>> And, for utilizing the efficient backend storage capabilities,
we
> can use
> >>>>>>>> the new storage snapshots API.
> >>>>>>>>
> >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi everyone,
> >>>>>>>>>
> >>>>>>>>> Whatever we do here, we need to have a plan to deal
with the fact
> >>>>>>>>> that we already have a feature (in 4.6 and later)
that allows
> you to use
> >>>>>>>>> the existing volume-snapshot APIs to create a volume
snapshot
> (for managed
> >>>>>>>>> storage) that resides on a backend SAN (using a
custom snapshot
> strategy
> >>>>>>>>> and a custom data motion strategy).
> >>>>>>>>>
> >>>>>>>>> If these new APIs go in, then how should the original
> >>>>>>>>> implementation (present in 4.6 and later) be changed?
If it is
> changed, how
> >>>>>>>>> do we support customers who are already using the
original
> volume-snapshot
> >>>>>>>>> API to take snapshots on a backend SAN?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mike
> >>>>>>>>>
> >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> >>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Will you be able to create a Template from a
StorageSnapshot?
> If
> >>>>>>>>>> yes, will the template be stored in the secondary
storage like
> normal
> >>>>>>>>>> templates or will that be handled somehow on
the vendor side?
> >>>>>>>>>>
> >>>>>>>>>> *Will STEVENS*
> >>>>>>>>>> Lead Developer
> >>>>>>>>>>
> >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
1S6
> >>>>>>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq
<
> >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks Will!!!
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens
<
> >>>>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I explicitly linked the Design Spec
in the Jira ticket because
> >>>>>>>>>>>> it was not clear in the 'mention' section
because it shows as
> a page 'you
> >>>>>>>>>>>> do not have permission to'.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Will STEVENS*
> >>>>>>>>>>>> Lead Developer
> >>>>>>>>>>>>
> >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
*|* H3J 1S6
> >>>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed
Ahmed <
> sahmed@cloudops.com
> >>>>>>>>>>>> > wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Design Spec:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jira Ticket
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We plan to propose a new set of
APIs to do snapshots on
> >>>>>>>>>>>>> managed storage backends like SolidFire.
Snapshots on
> current managed
> >>>>>>>>>>>>> storage stay on the
> >>>>>>>>>>>>> device which is contrary to what
CloudStack calls snpshots.
> >>>>>>>>>>>>> But taking snapshots
> >>>>>>>>>>>>> on storage and keeping it there
has its own advantages and we
> >>>>>>>>>>>>> would ideally like
> >>>>>>>>>>>>> to have both ways of doing snapshots.
This proposal adds 4
> new
> >>>>>>>>>>>>> APIs to
> >>>>>>>>>>>>> create snapshots on backend storage.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What do you guys think of this feature?
I would love to have
> >>>>>>>>>>>>> some feedback. I am working on making
the design spec more
> concrete but
> >>>>>>>>>>>>> wanted to have
> >>>>>>>>>>>>> a high level feedback first before
starting to work on it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> -Syed
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Mike Tutkowski*
> >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>> o: 303.746.7302
> >>>>>>>>> Advancing the way the world uses the cloud
> >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud
> >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the cloud
> >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the cloud
> >>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>
> >>
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message