cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Syed Ahmed <sah...@cloudops.com>
Subject Re: [PROPOSE] Backup and Recovery Framework
Date Mon, 11 Jun 2018 14:19:24 GMT
+1 for this feature Nicolas. As Will had mentioned, snapshots are not
backups. snapshots just ensure crash consistency but don’t necessarily
guarantee data consistency. My question is that whether there is going to
be a 3rd party backup solution integration that will serve as a reference?

Thanks,
-Syed
On Mon, Jun 11, 2018 at 5:57 AM Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> Thanks for the details Nicolas. I do think the picture is clear(er?) now.
> At least, now I understand what you are doing.
>
> BTW: It is a very interesting feature ;)
>
> On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
> > Thanks for clarifying Nicolas.  I just wanted to make sure that there
> > wasn't any confusion in this conversation as a 'snapshot' has different
> > meanings to different people/vendors, so I wanted to make sure we were
> all
> > on the same page.
> >
> > BTW, for what it is worth, I love this initiative.  This is an important
> > feature integration which I know a LOT of organizations will appreciate.
> >
> > Cheers,
> >
> > *Will Stevens*
> > Chief Technology Officer
> > c 514.826.0190
> >
> > <https://goo.gl/NYZ8KK>
> >
> >
> > On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> > > First of all, the framework is agnostic of any backup vendor/provider.
> It
> > > just provides the way to communicate with the provider to allow users
> to
> > > backup their VMs.
> > >
> > >
> > > Will is right, backups are not the same as snapshots, I got confused
> with
> > > the Veeam backup which starts from taking a snapshot to generate the
> > > backup. The way they are generated are up to each vendor, and it is not
> > > relevant to the framework. Sorry for the confusion, we are also
> > introducing
> > > the Veeam plugin in top of this framework.
> > >
> > >
> > > So, in terms of this framework, it does not matter what a backup is, we
> > > just keep a reference to an existing backup on each vendor.
> > >
> > >
> > > I would also like to state that this framework is not using a snapshot
> > > mechanism for backups in CloudStack. It is just providing interfaces
> and
> > > mechanisms for vendors/providers plugins in CloudStack. Implementation
> of
> > > each plugin should be relevant to the vendor and any backup handling
> will
> > > be external to CloudStack.
> > >
> > >
> > > I think that the diagram in the FS could explain what the task of the
> > > framework is:
> > > https://cwiki.apache.org/confluence/download/
> > attachments/85476500/Backup%20and%20Recovery%20UML.png?
> > version=1&modificationDate=1528367734183&api=v2.
> > > The framework simply allows management server to communicate with
> backup
> > > and recovery vendors/providers. To allow CloudStack communication with
> an
> > > specific vendor, a plugin should be implemented for it (as we are doing
> > > with Veeam for example).
> > >
> > >
> > > Please let me know if it is still confusing and I'll try to explain
> > better
> > >
> > > ________________________________
> > > From: Todd Pigram <todd@toddpigram.com>
> > > Sent: Thursday, June 7, 2018 12:11:18 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [PROPOSE] Backup and Recovery Framework
> > >
> > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use
> Alike
> > > for our XS pools. Inside ACS, the VMs use MaxBackup
> > >
> > >
> > > Nicolas.Vazquez@shapeblue.com
> > > www.shapeblue.com
> > > ,
> > > @shapeblue
> > >
> > >
> > >
> > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram <todd@toddpigram.com>
> > wrote:
> > >
> > > > I agree completely. That is why I use inside VM backup products
> > > currently.
> > > >
> > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > >> I have not been following this thread entirely, but I would not
> count
> > on
> > > >> our snapshot mechanism for backups.  If something like Veeam is
> used,
> > > then
> > > >> an agent is in play and that agent will quiesce the running
> processes,
> > > >> such
> > > >> as MySQL.  This is important in order to do backup and DR as you
> will
> > > lose
> > > >> transactions in memory and can potentially have a corrupted backup.
> > > >>
> > > >> Snapshots are not backups!!!
> > > >>
> > > >> *Will Stevens*
> > > >> Chief Technology Officer
> > > >> c 514.826.0190
> > > >>
> > > >> <https://goo.gl/NYZ8KK>
> > > >>
> > > >>
> > > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner <
> > > >> rafaelweingartner@gmail.com> wrote:
> > > >>
> > > >> > Ok, now I start to understand.  So, we will be handling snapshots
> > and
> > > >> they
> > > >> > will be sent to a third-party system.
> > > >> >
> > > >> > Now let’s proceed. If I configure this “feature” with
> > external=false.
> > > >> What
> > > >> > does it do? Nothing? I mean, what is the purpose of this
> “external”
> > > >> > parameter?
> > > >> >
> > > >> >
> > > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez <
> > > >> > Nicolas.Vazquez@shapeblue.com> wrote:
> > > >> >
> > > >> > > Its format may depend on the provider, but under the hood
it is
> a
> > > >> > snapshot
> > > >> > > which can be full or incremental and it is copied and stored
in
> a
> > > >> backup
> > > >> > > repository in one of the backup servers. For example, if
you use
> > > >> Veeam,
> > > >> > you
> > > >> > > will find your backups stored as .vbk (for full backups)
and
> .vib
> > > (for
> > > >> > > incremental backups). It is the task of the backup provider
to
> > > create
> > > >> and
> > > >> > > move these files to the backup repository.
> > > >> > >
> > > >> > > ________________________________
> > > >> > > From: Rafael Weingärtner <rafaelweingartner@gmail.com>
> > > >> > > Sent: Thursday, June 7, 2018 10:39:35 AM
> > > >> > > To: dev
> > > >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework
> > > >> > >
> > > >> > > Ok, there are a lot of things, which are kind of blurry
to me.
> So,
> > > >> let’s
> > > >> > > try to go step by step.
> > > >> > >
> > > >> > > Let me ask again. What is a backup of the VM? You mention
> > > >> restore-point.
> > > >> > > What is it? Is it a snapshot in the hypervisor (complete/partial
> > > >> > snapshot)?
> > > >> > > If it is not a snapshot,  what is it then? Are you installing
> some
> > > >> agent
> > > >> > in
> > > >> > > the VM and then copying/mirroring the files in the VM’s
file
> > system?
> > > >> > >
> > > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez <
> > > >> > > Nicolas.Vazquez@shapeblue.com> wrote:
> > > >> > >
> > > >> > > > Hi Daniel and Rafael,
> > > >> > > >
> > > >> > > >
> > > >> > > > I have updated the diagram picture on the FS, please
let me
> know
> > > if
> > > >> the
> > > >> > > > problem persists.
> > > >> > > >
> > > >> > > >
> > > >> > > > @<mailto:daniel.herrmann@zv.fraunhofer.de>Daniel
you are
> right
> > > >> about
> > > >> > > > restoring VMs, please let me update the FS. Regarding
your
> > > >> question, we
> > > >> > > > have considered the backups as a first class citizens,
meaning
> > > that
> > > >> a
> > > >> > > > backup would continue existing unless its originating
VM has
> > been
> > > >> > > > removed/expunged. This way, listBackups API method
would list
> > them
> > > >> in
> > > >> > > order
> > > >> > > > to allow a user to restore a deleted VM from a backup.
> > > >> > > >
> > > >> > > >
> > > >> > > > @Rafael the idea of the Dummy provider is simply returning
the
> > > >> expected
> > > >> > > > entities from a backup provider. For example, the
> BackupManager
> > > >> could
> > > >> > ask
> > > >> > > > the BackupProvider to list its backups and would expect
a list
> > of
> > > >> > > backups.
> > > >> > > > As the Dummy Provider implements the BackupProvider
interface,
> > it
> > > >> has
> > > >> > its
> > > >> > > > own implementation of listBackups() method and simply
returns
> > some
> > > >> mock
> > > >> > > > entities to the Backup Manager. It does not store any
backup,
> it
> > > >> simply
> > > >> > > > creates them in execution time and returns them.
> > > >> > > >
> > > >> > > >
> > > >> > > > From the CloudStack point of view, Backups are stored
> > externally,
> > > on
> > > >> > the
> > > >> > > > Backup and Recovery provider servers and are handled
by them.
> We
> > > >> keep
> > > >> > > track
> > > >> > > > of the existing backups in the Backup provider. Implementation
> > may
> > > >> > depend
> > > >> > > > on the Backup and Recovery provider, would consist
on
> snapshots,
> > > but
> > > >> > are
> > > >> > > > handled by the provider and not by CloudStack.
> > > >> > > >
> > > >> > > >
> > > >> > > > Backup policies are created on the Backup and Recovery
> provider
> > > and
> > > >> > > > provide retention options and periods as well as backup
times.
> > For
> > > >> > > example,
> > > >> > > > the provider may expose Golden, Silver and Bronze policies
and
> > > have
> > > >> > > > different options for each of them.
> > > >> > > >
> > > >> > > >
> > > >> > > > The 'external' parameter on the listBackupPolicies
API method
> > > would
> > > >> > only
> > > >> > > > ask the Backup and Recovery provider to list their
policies if
> > it
> > > is
> > > >> > set
> > > >> > > to
> > > >> > > > true. If it is not set, or set to false, it just simply
> returns
> > > the
> > > >> > > > existing mapped policies in CloudStack. Initially,
CloudStack
> > does
> > > >> not
> > > >> > > > store any backup policy, those should be imported/mapped
by
> the
> > > >> > > > 'importBackupPolicy' API method.
> > > >> > > >
> > > >> > > >
> > > >> > > > Finally, we consider backups as restore points. As
mentioned
> > > above,
> > > >> > > > backups are stored on the Backup and Recovery provider
> servers,
> > so
> > > >> we
> > > >> > > keep
> > > >> > > > a mapping between the CloudStack entity and the backup,
which
> > can
> > > be
> > > >> > used
> > > >> > > > to restore VMs.
> > > >> > > >
> > > >> > > >
> > > >> > > > Please let me know if anything.
> > > >> > > >
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > >
> > > >> > > > Nicolas Vazquez
> > > >> > > >
> > > >> > > > ________________________________
> > > >> > > > From: Rafael Weingärtner <rafaelweingartner@gmail.com>
> > > >> > > > Sent: Thursday, June 7, 2018 6:55:57 AM
> > > >> > > > To: dev
> > > >> > > > Subject: Re: [PROPOSE] Backup and Recovery Framework
> > > >> > > >
> > > >> > > > Hi Nicolas,
> > > >> > > >
> > > >> > > > The idea is great. I have some questions though.
> > > >> > > >
> > > >> > > > Before getting to the questions, your pictures are
broken, or
> at
> > > >> least
> > > >> > > they
> > > >> > > > seem broken to me.
> > > >> > > >
> > > >> > > > How would the “dummy” backup provider work? I mean,
what does
> it
> > > do?
> > > >> > > where
> > > >> > > > does it storage the "backup"?
> > > >> > > > What is the difference from this Backup to a Snapshot?
Is it
> > > simply
> > > >> > > storing
> > > >> > > > a snapshot outside the cloud structure?
> > > >> > > > How would these “backup policies” look like? I
mean, what can
> I
> > > >> > > configure?
> > > >> > > > Regarding that parameter “external” of your backup
policy, how
> > > >> would it
> > > >> > > > work if we set it to false?
> > > >> > > > What are you defining as a backup (what do you mean
by backup
> in
> > > the
> > > >> > > > context of this feature?)? I mean, is it simply a snapshot
> file
> > > that
> > > >> > you
> > > >> > > > store somewhere else?
> > > >> > > >
> > > >> > > >
> > > >> > > > Nicolas.Vazquez@shapeblue.com
> > > >> > > > www.shapeblue.com<http://www.shapeblue.com>
> > > >> > > > ,
> > > >> > > > @shapeblue
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > > Nicolas.Vazquez@shapeblue.com
> > > >> > > www.shapeblue.com<http://www.shapeblue.com>
> > > >> > > ,
> > > >> > > @shapeblue
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > On Thu, Jun 7, 2018 at 6:05 AM, <daniel.herrmann@zv.
> > fraunhofer.de
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi all,
> > > >> > > > >
> > > >> > > > > +1 for this feature request, thank you for bringing
this to
> > > life!
> > > >> > > Volume
> > > >> > > > > snapshots are what our users currently use as
backup
> solution,
> > > >> which
> > > >> > > has
> > > >> > > > > some inherent problems such as handling large
volumes and
> the
> > > >> > handling
> > > >> > > of
> > > >> > > > > multiple volumes (consistency).
> > > >> > > > >
> > > >> > > > > One thought: when restoring a VM, I think there
are two
> cases
> > to
> > > >> > > > consider:
> > > >> > > > >
> > > >> > > > > - The VM is still in CS and the user just wants
to restore
> an
> > > old
> > > >> > > version
> > > >> > > > > of the VM
> > > >> > > > > - The VM has already been deleted and the user
wants to
> > restore
> > > >> the
> > > >> > > > > deleted VM
> > > >> > > > >
> > > >> > > > > As listBackups is described as " listBackups API
method.
> List
> > > >> > existing
> > > >> > > > > backups for a VM". Is the second use case supported
at all?
> > > >> > > > >
> > > >> > > > > I like the approach not to define the backup policies
in CS
> > but
> > > in
> > > >> > > > > whatever tool is used in the end, as otherwise
a lot of
> > features
> > > >> > might
> > > >> > > be
> > > >> > > > > lost. We as services providers need to create
them in the
> > first
> > > >> > place,
> > > >> > > > but
> > > >> > > > > I think this is the better solution.
> > > >> > > > >
> > > >> > > > > Regards
> > > >> > > > > Daniel
> > > >> > > > >
> > > >> > > > > On 06.06.18, 14:43, "Nicolas Vazquez" <
> > > >> > Nicolas.Vazquez@shapeblue.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > >     Hi all,
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >     We would like to introduce a new framework
into
> CloudStack
> > > >> that
> > > >> > > will
> > > >> > > > > allow adding Backup and Recovery providers as
plugins. This
> > > >> framework
> > > >> > > > goal
> > > >> > > > > is allowing users to backup their guest VMs for
recovery
> > > purposes.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >     Please find the FS on this link:
> > https://cwiki.apache.org/
> > > >> > > > > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >     Regards,
> > > >> > > > >
> > > >> > > > >     Nicolas Vazquez
> > > >> > > > >
> > > >> > > > >     Nicolas.Vazquez@shapeblue.com
> > > >> > > > >     www.shapeblue.com<http://www.shapeblue.com>
> > > >> > > > >     ,
> > > >> > > > >     @shapeblue
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Rafael Weingärtner
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Rafael Weingärtner
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > Todd Pigram
> > > > http://about.me/ToddPigram
> > > > www.linkedin.com/in/toddpigram/<
> http://www.linkedin.com/in/toddpigram/
> > >
> > > > @pigram86 on twitter
> > > > https://plus.google.com/+ToddPigram86
> > > > Mobile - 216-224-5769
> > > >
> > > > PGP Public Key
> > > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > Todd Pigram
> > > http://about.me/ToddPigram
> > > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/
> >
> > > @pigram86 on twitter
> > > https://plus.google.com/+ToddPigram86
> > > Mobile - 216-224-5769
> > >
> > > PGP Public Key
> > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB>
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message