cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <williamstev...@gmail.com>
Subject Re: Modern template hosting
Date Fri, 03 Mar 2017 15:23:14 GMT
@rafael: in general, I don't think the client should ever be making a
choice about a mirror. We have to assume we are working with a scripted
application and anything hard coded in that implementation on the client
side is a risk.

I may not be understanding the use of the 300 approach, so I will research
it to see if I can make it fit.

On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <rafaelweingartner@gmail.com>
wrote:

Will great job.

I had the same doubt as Daan.

Have you considered using HTTP 300 working mode? Then, we could let the
client decide which mirror is the best(closest?). Section “10.3.1” of [1]
talks about it; this would put the pressure on deciding upon a mirror on
the client side, but I think it would be fair.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

PS: You really like very short variables names!

On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <daan.hoogland@shapeblue.com>
wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <swill@apache.org> wrote:
>
>     Hey All,
>     Please review this repo and tell me what you think:
>     https://github.com/swill/mirror302
>
>     Let me know if you have questions or you would like me to make changes
>     before I bring this topic up with ASF Infra and open a ticket to get
> this
>     implemented.
>
>     Cheers,
>
>     *Will Stevens*
>
>
>
>     On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
>     > I am building a short term solution right now.  I hopefully will
> have the
>     > building blocks in place by the weekend so we can start working with
> Infra
>     > to get it in place.
>     >
>     > We will have a web server which we can point something like
>     > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
>     > repo which tracks a mirror list and exposes a static site to be able
> to
>     > query the mirror list.
>     >
>     > The web server will receive a request and will do a 302 redirect to
> the
>     > appropriate resource on one of the mirrors.
>     >
>     > This gets us started.  Legacy environments will have to do a DB
> change to
>     > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
>     > whatever), but otherwise it should be pretty simple.
>     >
>     > We can improve how we deliver templates going forward, but this
> obviously
>     > requires some discussion still.  I will try to buy us some time with
> an
>     > implementation which solves our problems today...
>     >
>     > *Will STEVENS*
>     > Lead Developer
>     >
>     > <https://goo.gl/NYZ8KK>
>     >
>     > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>     > abhinandan.prateek@shapeblue.com> wrote:
>     >
>     >>
>     >> Initial seeding is a manual step and that is not going to change in
> near
>     >> future. A handy list of official places from where these templates
> can be
>     >> downloaded won’t harm this part of the installation. Just allow
> anyone
>     >> credible who follows democratic process to publish their templates
> with
>     >> apache’s blessings.
>     >>
>     >>
>     >> Coming to urls hard coded in db, the popular suggestion is to point
> them
>     >> to mirrors. This looks good but this is going to take some effort
> that has
>     >> not materialised yet. Once this change is made it will be going in
> some
>     >> future release like 4.11; the problem still remains for the
> releases that
>     >> are already out there. In case access to download.cloud.com is
> dropped
>     >> at some point we will have to document the procedure such that
> people know
>     >> how to make these stable releases work. That is where the procedure
>     >> documented here https://shankerbalan.net/blog/
>     >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
>     >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
>     >> script to make choices instead of written wiki. Note that this will
> be
>     >> anyway needed by the older releases.
>     >>
>     >>
>     >>
>     >>
>     >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.angus@shapeblue.com>
> wrote:
>     >>
>     >> >-1
>     >> >
>     >> >We are trying to get MORE people to use CloudStack.  And raising
> the
>     >> barrier to entry does nothing to help that.
>     >> >
>     >> >
>     >> >Kind regards,
>     >> >
>     >> >Paul Angus
>     >> >
>     >> >paul.angus@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> >@shapeblue
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >-----Original Message-----
>     >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>     >> >Sent: 01 March 2017 09:39
>     >> >To: dev@cloudstack.apache.org
>     >> >Subject: Re: Modern template hosting
>     >> >
>     >> >+1
>     >> >
>     >> >After seeing this discussion running in circles several times, I
> think
>     >> we should at least get started with the simplest option.
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chiradeepv@gmail.com>
> wrote:
>     >> >
>     >> >>The alternative foolproof way is to simply not provide the
> automatic
>     >> >>download.
>     >> >>
>     >> >>Just document the mirror list URL where the templates can be
> found. I
>     >> >>don't know why people reject this option.
>     >> >>
>     >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>     >> >>
>     >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>     >> >><williamstevens@gmail.com>
>     >> >>wrote:
>     >> >>
>     >> >>> OK. Thanks for the heads up.
>     >> >>>
>     >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
>     >> wrote:
>     >> >>>
>     >> >>> > Sounds workable. The downloader code in the SSVM won't
follow
>     >> >>> > redirects I think.
>     >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
>     >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>     >> >>> > HttpTemplateDownloader.java#L93
>     >> >>> > https://goo.gl/dSi0r5
>     >> >>> >  Might need to add
>     >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>     >> >>> >
>     >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>     >> >>> > <wstevens@cloudops.com>
>     >> >>> > wrote:
>     >> >>> >
>     >> >>> > > We haven't opened a ticket yet because we don't have
a
> strategy
>     >> yet.
>     >> >>> > >
>     >> >>> > > What do you guys think of this:
>     >> >>> > > - We setup a new github repo in the 'apache' org
which
> consists
>     >> >>> > > of a
>     >> >>> > single
>     >> >>> > > file with a list of active/supported mirrors.
>     >> >>> > > - I write a small web server, distributed as a binary,
> which can
>     >> >>> > > be
>     >> >>> > hosted
>     >> >>> > > by ASF Infra.  This web server will query the current
list
> of
>     >> >>> > > mirrors
>     >> >>> and
>     >> >>> > > will select one and then do a 302 redirect to that
mirror.
>     >> >>> > >
>     >> >>> > > The act of 'choosing' a mirror could be done in a
number of
> ways.
>     >> >>> > > - If we want to define an order, then it could just
try
> from the
>     >> >>> > > top of
>     >> >>> > the
>     >> >>> > > list and work its way down.  It would curl the target
to
> make
>     >> >>> > > sure it
>     >> >>> > gets
>     >> >>> > > a 200 and if it does, it would do a 302 redirect.
>     >> >>> > > - Or, if we want to distribute the load across the
mirrors,
> we
>     >> >>> > > could
>     >> >>> pick
>     >> >>> > > from the list randomly.  Again, doing a curl to verify
the
> mirror
>     >> >>> > > is up
>     >> >>> > and
>     >> >>> > > then doing a redirect.
>     >> >>> > > - If we want to get fancy, we could do a reverse
IP lookup
> and
>     >> >>> > > try to
>     >> >>> > match
>     >> >>> > > the requester with their closest geographical mirror.
>     >> >>> > >
>     >> >>> > > Thoughts?
>     >> >>> > >
>     >> >>> > > *Will STEVENS*
>     >> >>> > > Lead Developer
>     >> >>> > >
>     >> >>> > > <https://goo.gl/NYZ8KK>
>     >> >>> > >
>     >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
<
>     >> >>> chiradeepv@gmail.com
>     >> >>> > >
>     >> >>> > > wrote:
>     >> >>> > >
>     >> >>> > > > What steps are needed to set up a mirror? What
does Infra
> need
>     >> to do?
>     >> >>> > Has
>     >> >>> > > > anybody filed a ticket with Infra?
>     >> >>> > > >
>     >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela
<
>     >> >>> > > > raja.pullela@accelerite.com>
>     >> >>> > > > wrote:
>     >> >>> > > >
>     >> >>> > > > > Hi will,
>     >> >>> > > > >
>     >> >>> > > > > I believe, we didn’t get to close ‘getting
a mirror on
> Apache’
>     >> >>> > because
>     >> >>> > > we
>     >> >>> > > > > needed someone on the Apache Infra side
to close this.
> BTW,
>     >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages
this?) has
>     >> >>> > > > > all/most of
>     >> >>> > the
>     >> >>> > > > > content.  Once we can close on the Apache
mirror for
> hosting
>     >> >>> > > > > the
>     >> >>> > > > content, I
>     >> >>> > > > > can help assist getting the content there.
>     >> >>> > > > >
>     >> >>> > > > > For now, we have replicated the download.cloud.com
> content to
>     >> >>> > > > > ‘ s3.download.accelerite.com’.
>     >> >>> > > > > Also, we are working on a set of steps/procedure
to
> help with
>     >> >>> > > > > this change.  I will update everyone in
about a week’s
> time
>     >> >>> > > > > on the
>     >> >>> > details.
>     >> >>> > > > >
>     >> >>> > > > > Best,
>     >> >>> > > > > Raja Pullela
>     >> >>> > > > > Engineering Team,
>     >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa
Clara, CA,
95054
>     >> >>> > > > >
>     >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on
> behalf of
>     >> >>> > > > > Will Stevens" <williamstevens@gmail.com
on behalf of
>     >> >>> > wstevens@cloudops.com>
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > unfortunately the template mirror conversation
got
> caught up
>     >> >>> > > > > in
>     >> >>> > details
>     >> >>> > > > and
>     >> >>> > > > > nobody took the lead on implementing a
solution.
>     >> >>> > > > >
>     >> >>> > > > > citrix has been pinging me every couple
months to say
> 'dude,
>     >> >>> > > > > we
>     >> >>> need
>     >> >>> > to
>     >> >>> > > > > remove the dependency on download.citrix.com',
but i
> have not
>     >> >>> > > > > had
>     >> >>> > the
>     >> >>> > > > > cycles to get in and solve the problem.
 the shutdown
> of that
>     >> >>> > > > > is
>     >> >>> > > imminent
>     >> >>> > > > > right now, so we need to solve it asap.
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > *Will STEVENS*
>     >> >>> > > > > Lead Developer
>     >> >>> > > > >
>     >> >>> > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > >
>     >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul
Angus <
>     >> >>> > paul.angus@shapeblue.com
>     >> >>> > > >
>     >> >>> > > > > wrote:
>     >> >>> > > > >
>     >> >>> > > > > > Hi Nathan,
>     >> >>> > > > > >
>     >> >>> > > > > > Ideally, if you put the template location
in (or use
a
>     >> >>> > > > > > template
>     >> >>> > > defined
>     >> >>> > > > > > in)  test_data.py then the actual
location can be
>     >> >>> > > > > > overridden by
>     >> >>> > > anyone
>     >> >>> > > > > > testing.
>     >> >>> > > > > >
>     >> >>> > > > > > For Trillian, we've copied all of
the templates that
> people
>     >> >>> > > > > > have
>     >> >>> > > define
>     >> >>> > > > > to
>     >> >>> > > > > > a local repo and then replace the
URLs in
> test_data.py to
>     >> >>> > > > > > reduce
>     >> >>> > > > > bandwidth
>     >> >>> > > > > > use and download times.
>     >> >>> > > > > >
>     >> >>> > > > > > Ie:
>     >> >>> > > > > >
>     >> >>> > > > > >             "bootableIso":
>     >> >>> > > > > >                 {
>     >> >>> > > > > >                     "displaytext":
"Test Bootable
> ISO",
>     >> >>> > > > > >                     "name": "testISO",
>     >> >>> > > > > >                     "bootable": True,
>     >> >>> > > > > >                     "ispublic": False,
>     >> >>> > > > > >                     "url": "{{ marvin_images_location
>     >> >>> > > > > > }}/TinyCore-current.iso",
>     >> >>> > > > > >                     "ostype": 'Other
Linux (64-bit)',
>     >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>     >> >>> > > > > >         },
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > I thought that we had come up with
a solution for
>     >> >>> > download.cloud.com
>     >> >>> > > ,
>     >> >>> > > > by
>     >> >>> > > > > > having a mirrorlist hosted in Community
Apache
> 'space' with
>     >> >>> anyone
>     >> >>> > > able
>     >> >>> > > > > to
>     >> >>> > > > > > out themselves forward as a mirror.
>     >> >>> > > > > > But I must admit I lost track of whether
anyone made
> the
>     >> >>> requisite
>     >> >>> > > > > changes
>     >> >>> > > > > > in code....
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > Kind regards,
>     >> >>> > > > > >
>     >> >>> > > > > > Paul Angus
>     >> >>> > > > > >
>     >> >>> > > > > > paul.angus@shapeblue.com
>     >> >>> > > > > > www.shapeblue.com
>     >> >>> > > > > > 53 Chandos Place, Covent Garden, London
 WC2N 4HSUK
>     >> >>> > > > > > @shapeblue
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > -----Original Message-----
>     >> >>> > > > > > From: williamstevens@gmail.com
>     >> >>> > > > > > [mailto:williamstevens@gmail.com]
>     >> >>> > On
>     >> >>> > > > > > Behalf Of Will Stevens
>     >> >>> > > > > > Sent: 24 February 2017 16:30
>     >> >>> > > > > > To: dev@cloudstack.apache.org
>     >> >>> > > > > > Subject: Re: Modern template hosting
>     >> >>> > > > > >
>     >> >>> > > > > > this is a hard questions.  in general,
we should be
> setting
>     >> >>> > > > > > up a
>     >> >>> > > mirror
>     >> >>> > > > > on
>     >> >>> > > > > > some cloudstack/apache domain and
then mirror to
other
>     >> >>> > > > > > provided
>     >> >>> > > > > templates.
>     >> >>> > > > > >
>     >> >>> > > > > > we MUST come up with a solution to
deprecate
>     >> >>> > > > > > 'download.cloud.com
>     >> >>> ',
>     >> >>> > > > that
>     >> >>> > > > > > is going to be going away any day
now.
>     >> >>> > > > > >
>     >> >>> > > > > > i don't know the right way to solve
this to be
> honest, but
>     >> >>> > > > > > if you
>     >> >>> > > have
>     >> >>> > > > > > ideas, i am willing to help.
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > >
>     >> >>> > > > > > *Will STEVENS*
>     >> >>> > > > > > Lead Developer
>     >> >>> > > > > >
>     >> >>> > > > > > <https://goo.gl/NYZ8KK>
>     >> >>> > > > > >
>     >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM,
Nathan Johnson <
>     >> >>> njohnson@ena.com
>     >> >>> > >
>     >> >>> > > > > wrote:
>     >> >>> > > > > >
>     >> >>> > > > > > > So not to re-open a can of worms,
but I’m in a
> situation
>     >> >>> > > > > > > where
>     >> >>> I
>     >> >>> > > need
>     >> >>> > > > > > > to come up with a Marvin component
test that
> depends on a
>     >> >>> > template
>     >> >>> > > > > > > based on a kernel that’s relatively
new, i.e.,
> newer than
>     >> >>> Centos
>     >> >>> > > 5.3
>     >> >>> > > > /
>     >> >>> > > > > > Ubuntu 10.04 .
>     >> >>> > > > > > > I see openvm.eu has a suitable
template (Ubuntu
> 16.0.4
>     >> >>> > > > > > > for
>     >> >>> KVM),
>     >> >>> > > but
>     >> >>> > > > > > > from looking at the thread "Migrating
CloudStack
> content
>     >> >>> > > > > > > from download.cloud.com” it
looks like there is
>     >> >>> > > > > > > resistance to using
>     >> >>> > > this
>     >> >>> > > > at
>     >> >>> > > > > > > least for hosting system vm templates
over concerns
> of
>     >> >>> > neutrality.
>     >> >>> > > > > > > Would this be suitable for a
component test?  If
> not,
>     >> >>> > > > > > > what is a
>     >> >>> > > > > “blessed”
>     >> >>> > > > > > > template location?
>     >> >>> > > > > > >
>     >> >>> > > > > > > Thanks in advance!
>     >> >>> > > > > > >
>     >> >>> > > > > > > Nathan Johnson
>     >> >>> > > > > > > R&D Engineer
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > > 618 Grassmere Park Drive, Suite
12 Nashville, TN
> 37211
>     >> >>> > > > > > > General Office: 615-312-6000
>     >> >>> > > > > > >
>     >> >>> > > > > > > website | blog | support
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > > >
>     >> >>> > > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > >
>     >> >>> > > > > DISCLAIMER
>     >> >>> > > > > ==========
>     >> >>> > > > > This e-mail may contain privileged and
confidential
>     >> >>> > > > > information
>     >> >>> which
>     >> >>> > > is
>     >> >>> > > > > the property of Accelerite, a Persistent
Systems
> business. It
>     >> >>> > > > > is
>     >> >>> > > intended
>     >> >>> > > > > only for the use of the individual or entity
to which
> it is
>     >> >>> > addressed.
>     >> >>> > > If
>     >> >>> > > > > you are not the intended recipient, you
are not
> authorized to
>     >> >>> > > > > read,
>     >> >>> > > > retain,
>     >> >>> > > > > copy, print, distribute or use this message.
If you
have
>     >> >>> > > > > received
>     >> >>> > this
>     >> >>> > > > > communication in error, please notify the
sender and
> delete
>     >> >>> > > > > all
>     >> >>> > copies
>     >> >>> > > of
>     >> >>> > > > > this message. Accelerite, a Persistent
Systems business
> does
>     >> >>> > > > > not
>     >> >>> > accept
>     >> >>> > > > any
>     >> >>> > > > > liability for virus infected mails.
>     >> >>> > > > >
>     >> >>> > > >
>     >> >>> > >
>     >> >>> >
>     >> >>>
>     >> >
>     >> >abhinandan.prateek@shapeblue.com
>     >> >www.shapeblue.com
>     >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>     >> >
>     >> >
>     >> >
>     >>
>     >> abhinandan.prateek@shapeblue.com
>     >> www.shapeblue.com
>     >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     >> @shapeblue
>     >>
>     >>
>     >>
>     >>
>     >
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>


--
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message