cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: State of the S3 secondary storage
Date Tue, 19 Jul 2016 20:24:45 GMT
Oh ya.  Gotta love NFS syncing to servers halfway around the world.  :P

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sahmed@cloudops.com> wrote:

> Right, but is that an artificial limitation imposed by Cloudstack of is
> there something that will break if we make NFS region wide? If not, we can
> look at moving the NFS to a region wide storage.
>
> -Syed
>
> On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
> > Probably because ACS does not treat it as region wide, unlike the object
> > storage implementations.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sahmed@cloudops.com> wrote:
> >
> > > Is there a reason why NFS cannot be region-wide?
> > >
> > > -Syed
> > >
> > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nux@li.nux.ro> wrote:
> > >
> > > > Thanks for your input guys.
> > > >
> > > > I think I'll stay with NFS for now after your replies, but it'd be
> > pretty
> > > > sweet the day when we'll be having region wide secondary storage.
> > > >
> > > > I'll need to think of another way to provide DR, perhaps some
> automatic
> > > > snapshot+download of sorts.
> > > >
> > > > Lucian
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > ----- Original Message -----
> > > > > From: "Syed Ahmed" <sahmed@cloudops.com>
> > > > > To: dev@cloudstack.apache.org
> > > > > Cc: "Nux!" <nux@li.nux.ro>, "Cloudstack Users List" <
> > > > users@cloudstack.apache.org>
> > > > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > > > Subject: Re: State of the S3 secondary storage
> > > >
> > > > > As Will put it, the implementation for using Object store as
> > secondary
> > > > > storage relies on using Staging NFS. You would also loose the
> > > > functionality
> > > > > to do differential snapshots. All your snapshots will basically be
> > full
> > > > > volume snapshots. So, if you have a lot of snapshots being taken
at
> > > > > scheduled intervals, you might end up with wasting a lot of space
> on
> > > your
> > > > > S3. Not to mention all of this has to go through the NFS which can
> > (an
> > > in
> > > > > our experience has) fill up preventing you from creating new VMs
> > > (because
> > > > > templates cannot be downloaded). However you do benefit from a
> region
> > > > wide
> > > > > Image store. We've worked around the NFS problem by basically
> having
> > > > > multiple staging NFS stores.
> > > > >
> > > > > I have plans to make Object stores better integrated in Cloudstack.
> > The
> > > > > holy grail would be to avoid staging NFS completely (or atleast in
> > the
> > > > > upload path). But for now we have to live with the limitations.
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
> > wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > >> We are using Swift as secondary storage.  It also offers
> inter-zone
> > > > >> secondary storage.  We have had to do fixes to make it work
> > correctly,
> > > > but
> > > > >> we have pushed those fixes upstream so they are available in
4.9
> > (and
> > > > some
> > > > >> fixes earlier than that).
> > > > >>
> > > > >> Like Wido said, the staging NFS is still needed which is a real
> > > problem
> > > > >> with these implementations.  You end up copying the data many
> times
> > > and
> > > > you
> > > > >> will run into problems if you have lots of customers doing
> scheduled
> > > > >> snapshots at the same time because you can fill your NFS staging
> > area
> > > > >> unless you give it a lot of space to work with.
> > > > >>
> > > > >> The object storage integrations are working, but they are not
> > designed
> > > > very
> > > > >> well at present.  You have code for both S3 and Swift basically
> just
> > > > >> separated with conditionals and much of the logic is shared.
 This
> > is
> > > > very
> > > > >> brittle because making changes in one integration can (and
> > > historically
> > > > >> has) broken the other implementation.  This is a relatively big
> > > problem
> > > > >> because most people working with either Swift or S3 will only
have
> > the
> > > > >> ability to test one of them, so they don't really have the ability
> > to
> > > > test
> > > > >> if they have broken the other.  It does not help that there is
> Zero
> > CI
> > > > >> coverage run for either the Swift or S3 integrations due to the
> > > > difficulty
> > > > >> building and tearing down object storage environments on the
fly.
> > > > >>
> > > > >> All in all, yes they should be working, but expect a bit of a
> bumpy
> > > > road.
> > > > >>
> > > > >> Hope that helps,
> > > > >>
> > > > >> *Will STEVENS*
> > > > >> Lead Developer
> > > > >>
> > > > >> *CloudOps* *| *Cloud Solutions Experts
> > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > >> w cloudops.com *|* tw @CloudOps_
> > > > >>
> > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
> > wido@widodh.nl>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nux@li.nux.ro>:
> > > > >> > >
> > > > >> > >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > Could anyone summarise the state of S3 (and clones)
for
> > secondary
> > > > >> > storage?
> > > > >> > > I read at some point that it's the only secondary storage
type
> > > > >> supported
> > > > >> > inter-zone, so it would appeal from a DR pov, but not sure
how
> > > mature
> > > > it
> > > > >> is
> > > > >> > or if there are any serious gotchas involved.
> > > > >> > >
> > > > >> >
> > > > >> > We have it running for our largest region (Amsterdam) with
> Ceph's
> > > > RADOS
> > > > >> > Gateway as backend. We had some issues, but they seem to
be
> fixed
> > > > after
> > > > >> our
> > > > >> > latest patches.
> > > > >> >
> > > > >> > You still need the staging NFS though, so that doesn't solve
> > > anything.
> > > > >> >
> > > > >> > Wido
> > > > >> >
> > > > >> > > Thanks
> > > > >> > > Lucian
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > > >> > >
> > > > >> > > Nux!
> > > > >> > > www.nux.ro
> > > > >> >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message