Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
MIME-Version: 1.0
In-Reply-To: <CAC2pano5WWQeOpa1HjLwd7SS6Rt-T3y0EHWnPVLmXm0+9QOrKQ@mail.gmail.com>
References: <1474354026.4435.1468931915063.JavaMail.zimbra@li.nux.ro>
 <567675001.1239.1468942011256@ox.pcextreme.nl> <CAC2panoBnW2BeCS0F5PahWDwXbXv0Jw3Pg8VO=03qadMx=+-aw@mail.gmail.com>
 <CADMzvUpzkDoMzFz_ztyxxjmbRLKaGWRYBJBMqae_xUBMFu37hg@mail.gmail.com>
 <1479696618.4912.1468951692141.JavaMail.zimbra@li.nux.ro> <CADMzvUrt7z887o3eT_Kt-5=vv0HNOfQ5B6UjSezMXVzbcH3W8w@mail.gmail.com>
 <CAC2pano5WWQeOpa1HjLwd7SS6Rt-T3y0EHWnPVLmXm0+9QOrKQ@mail.gmail.com>
From: Syed Ahmed <sahmed@cloudops.com>
Date: Tue, 19 Jul 2016 16:06:35 -0400
Message-ID: <CADMzvUpPUW37LvY9YL35a3aDDWK=1aRvY9eOrBEozQqP=N7XbA@mail.gmail.com>
Subject: Re: State of the S3 secondary storage
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Cc: "Nux!" <nux@li.nux.ro>, Cloudstack Users List <users@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c0ba74839455a0538029f91
archived-at: Tue, 19 Jul 2016 20:06:44 -0000

--94eb2c0ba74839455a0538029f91
Content-Type: text/plain; charset=UTF-8

Right, but is that an artificial limitation imposed by Cloudstack of is
there something that will break if we make NFS region wide? If not, we can
look at moving the NFS to a region wide storage.

-Syed

On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <wstevens@cloudops.com> wrote:

> Probably because ACS does not treat it as region wide, unlike the object
> storage implementations.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sahmed@cloudops.com> wrote:
>
> > Is there a reason why NFS cannot be region-wide?
> >
> > -Syed
> >
> > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nux@li.nux.ro> wrote:
> >
> > > Thanks for your input guys.
> > >
> > > I think I'll stay with NFS for now after your replies, but it'd be
> pretty
> > > sweet the day when we'll be having region wide secondary storage.
> > >
> > > I'll need to think of another way to provide DR, perhaps some automatic
> > > snapshot+download of sorts.
> > >
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Syed Ahmed" <sahmed@cloudops.com>
> > > > To: dev@cloudstack.apache.org
> > > > Cc: "Nux!" <nux@li.nux.ro>, "Cloudstack Users List" <
> > > users@cloudstack.apache.org>
> > > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > > Subject: Re: State of the S3 secondary storage
> > >
> > > > As Will put it, the implementation for using Object store as
> secondary
> > > > storage relies on using Staging NFS. You would also loose the
> > > functionality
> > > > to do differential snapshots. All your snapshots will basically be
> full
> > > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > > scheduled intervals, you might end up with wasting a lot of space on
> > your
> > > > S3. Not to mention all of this has to go through the NFS which can
> (an
> > in
> > > > our experience has) fill up preventing you from creating new VMs
> > (because
> > > > templates cannot be downloaded). However you do benefit from a region
> > > wide
> > > > Image store. We've worked around the NFS problem by basically having
> > > > multiple staging NFS stores.
> > > >
> > > > I have plans to make Object stores better integrated in Cloudstack.
> The
> > > > holy grail would be to avoid staging NFS completely (or atleast in
> the
> > > > upload path). But for now we have to live with the limitations.
> > > >
> > > > -Syed
> > > >
> > > >
> > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
> wstevens@cloudops.com>
> > > > wrote:
> > > >
> > > >> We are using Swift as secondary storage.  It also offers inter-zone
> > > >> secondary storage.  We have had to do fixes to make it work
> correctly,
> > > but
> > > >> we have pushed those fixes upstream so they are available in 4.9
> (and
> > > some
> > > >> fixes earlier than that).
> > > >>
> > > >> Like Wido said, the staging NFS is still needed which is a real
> > problem
> > > >> with these implementations.  You end up copying the data many times
> > and
> > > you
> > > >> will run into problems if you have lots of customers doing scheduled
> > > >> snapshots at the same time because you can fill your NFS staging
> area
> > > >> unless you give it a lot of space to work with.
> > > >>
> > > >> The object storage integrations are working, but they are not
> designed
> > > very
> > > >> well at present.  You have code for both S3 and Swift basically just
> > > >> separated with conditionals and much of the logic is shared.  This
> is
> > > very
> > > >> brittle because making changes in one integration can (and
> > historically
> > > >> has) broken the other implementation.  This is a relatively big
> > problem
> > > >> because most people working with either Swift or S3 will only have
> the
> > > >> ability to test one of them, so they don't really have the ability
> to
> > > test
> > > >> if they have broken the other.  It does not help that there is Zero
> CI
> > > >> coverage run for either the Swift or S3 integrations due to the
> > > difficulty
> > > >> building and tearing down object storage environments on the fly.
> > > >>
> > > >> All in all, yes they should be working, but expect a bit of a bumpy
> > > road.
> > > >>
> > > >> Hope that helps,
> > > >>
> > > >> *Will STEVENS*
> > > >> Lead Developer
> > > >>
> > > >> *CloudOps* *| *Cloud Solutions Experts
> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > >> w cloudops.com *|* tw @CloudOps_
> > > >>
> > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
> wido@widodh.nl>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nux@li.nux.ro>:
> > > >> > >
> > > >> > >
> > > >> > > Hello,
> > > >> > >
> > > >> > > Could anyone summarise the state of S3 (and clones) for
> secondary
> > > >> > storage?
> > > >> > > I read at some point that it's the only secondary storage type
> > > >> supported
> > > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> > mature
> > > it
> > > >> is
> > > >> > or if there are any serious gotchas involved.
> > > >> > >
> > > >> >
> > > >> > We have it running for our largest region (Amsterdam) with Ceph's
> > > RADOS
> > > >> > Gateway as backend. We had some issues, but they seem to be fixed
> > > after
> > > >> our
> > > >> > latest patches.
> > > >> >
> > > >> > You still need the staging NFS though, so that doesn't solve
> > anything.
> > > >> >
> > > >> > Wido
> > > >> >
> > > >> > > Thanks
> > > >> > > Lucian
> > > >> > >
> > > >> > > --
> > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > >
> > > >> > > Nux!
> > > >> > > www.nux.ro
> > > >> >
> > >
> >
>

--94eb2c0ba74839455a0538029f91--