cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <sw...@apache.org>
Subject Re: Modern template hosting
Date Thu, 02 Mar 2017 23:23:01 GMT
Hey All,
Please review this repo and tell me what you think:
https://github.com/swill/mirror302

Let me know if you have questions or you would like me to make changes
before I bring this topic up with ASF Infra and open a ticket to get this
implemented.

Cheers,

*Will Stevens*



On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstevens@cloudops.com> wrote:

> I am building a short term solution right now.  I hopefully will have the
> building blocks in place by the weekend so we can start working with Infra
> to get it in place.
>
> We will have a web server which we can point something like
> downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
> repo which tracks a mirror list and exposes a static site to be able to
> query the mirror list.
>
> The web server will receive a request and will do a 302 redirect to the
> appropriate resource on one of the mirrors.
>
> This gets us started.  Legacy environments will have to do a DB change to
> change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> whatever), but otherwise it should be pretty simple.
>
> We can improve how we deliver templates going forward, but this obviously
> requires some discussion still.  I will try to buy us some time with an
> implementation which solves our problems today...
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> abhinandan.prateek@shapeblue.com> wrote:
>
>>
>> Initial seeding is a manual step and that is not going to change in near
>> future. A handy list of official places from where these templates can be
>> downloaded won’t harm this part of the installation. Just allow anyone
>> credible who follows democratic process to publish their templates with
>> apache’s blessings.
>>
>>
>> Coming to urls hard coded in db, the popular suggestion is to point them
>> to mirrors. This looks good but this is going to take some effort that has
>> not materialised yet. Once this change is made it will be going in some
>> future release like 4.11; the problem still remains for the releases that
>> are already out there. In case access to download.cloud.com is dropped
>> at some point we will have to document the procedure such that people know
>> how to make these stable releases work. That is where the procedure
>> documented here https://shankerbalan.net/blog/
>> seed-cloudstack-templates-offline/ should be officially adapted with a
>> marketplace to pick up templates of choosing. Maybe we can have a pre-setup
>> script to make choices instead of written wiki. Note that this will be
>> anyway needed by the older releases.
>>
>>
>>
>>
>> On 01/03/17, 4:24 PM, "Paul Angus" <paul.angus@shapeblue.com> wrote:
>>
>> >-1
>> >
>> >We are trying to get MORE people to use CloudStack.  And raising the
>> barrier to entry does nothing to help that.
>> >
>> >
>> >Kind regards,
>> >
>> >Paul Angus
>> >
>> >paul.angus@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> >@shapeblue
>> >
>> >
>> >
>> >
>> >-----Original Message-----
>> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
>> >Sent: 01 March 2017 09:39
>> >To: dev@cloudstack.apache.org
>> >Subject: Re: Modern template hosting
>> >
>> >+1
>> >
>> >After seeing this discussion running in circles several times, I think
>> we should at least get started with the simplest option.
>> >
>> >
>> >
>> >
>> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chiradeepv@gmail.com> wrote:
>> >
>> >>The alternative foolproof way is to simply not provide the automatic
>> >>download.
>> >>
>> >>Just document the mirror list URL where the templates can be found. I
>> >>don't know why people reject this option.
>> >>
>> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>> >>
>> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> >><williamstevens@gmail.com>
>> >>wrote:
>> >>
>> >>> OK. Thanks for the heads up.
>> >>>
>> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chiradeepv@gmail.com>
>> wrote:
>> >>>
>> >>> > Sounds workable. The downloader code in the SSVM won't follow
>> >>> > redirects I think.
>> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>> >>> > HttpTemplateDownloader.java#L93
>> >>> > https://goo.gl/dSi0r5
>> >>> >  Might need to add
>> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>> >>> >
>> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
>> >>> > <wstevens@cloudops.com>
>> >>> > wrote:
>> >>> >
>> >>> > > We haven't opened a ticket yet because we don't have a strategy
>> yet.
>> >>> > >
>> >>> > > What do you guys think of this:
>> >>> > > - We setup a new github repo in the 'apache' org which consists
>> >>> > > of a
>> >>> > single
>> >>> > > file with a list of active/supported mirrors.
>> >>> > > - I write a small web server, distributed as a binary, which
can
>> >>> > > be
>> >>> > hosted
>> >>> > > by ASF Infra.  This web server will query the current list
of
>> >>> > > mirrors
>> >>> and
>> >>> > > will select one and then do a 302 redirect to that mirror.
>> >>> > >
>> >>> > > The act of 'choosing' a mirror could be done in a number of
ways.
>> >>> > > - If we want to define an order, then it could just try from
the
>> >>> > > top of
>> >>> > the
>> >>> > > list and work its way down.  It would curl the target to make
>> >>> > > sure it
>> >>> > gets
>> >>> > > a 200 and if it does, it would do a 302 redirect.
>> >>> > > - Or, if we want to distribute the load across the mirrors,
we
>> >>> > > could
>> >>> pick
>> >>> > > from the list randomly.  Again, doing a curl to verify the
mirror
>> >>> > > is up
>> >>> > and
>> >>> > > then doing a redirect.
>> >>> > > - If we want to get fancy, we could do a reverse IP lookup
and
>> >>> > > try to
>> >>> > match
>> >>> > > the requester with their closest geographical mirror.
>> >>> > >
>> >>> > > Thoughts?
>> >>> > >
>> >>> > > *Will STEVENS*
>> >>> > > Lead Developer
>> >>> > >
>> >>> > > <https://goo.gl/NYZ8KK>
>> >>> > >
>> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>> >>> chiradeepv@gmail.com
>> >>> > >
>> >>> > > wrote:
>> >>> > >
>> >>> > > > What steps are needed to set up a mirror? What does Infra
need
>> to do?
>> >>> > Has
>> >>> > > > anybody filed a ticket with Infra?
>> >>> > > >
>> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
>> >>> > > > raja.pullela@accelerite.com>
>> >>> > > > wrote:
>> >>> > > >
>> >>> > > > > Hi will,
>> >>> > > > >
>> >>> > > > > I believe, we didn’t get to close ‘getting a
mirror on Apache’
>> >>> > because
>> >>> > > we
>> >>> > > > > needed someone on the Apache Infra side to close
this. BTW,
>> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?)
has
>> >>> > > > > all/most of
>> >>> > the
>> >>> > > > > content.  Once we can close on the Apache mirror
for hosting
>> >>> > > > > the
>> >>> > > > content, I
>> >>> > > > > can help assist getting the content there.
>> >>> > > > >
>> >>> > > > > For now, we have replicated the download.cloud.com
content to
>> >>> > > > > ‘ s3.download.accelerite.com’.
>> >>> > > > > Also, we are working on a set of steps/procedure
to help with
>> >>> > > > > this change.  I will update everyone in about a
week’s time
>> >>> > > > > on the
>> >>> > details.
>> >>> > > > >
>> >>> > > > > Best,
>> >>> > > > > Raja Pullela
>> >>> > > > > Engineering Team,
>> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA,
95054
>> >>> > > > >
>> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on behalf of
>> >>> > > > > Will Stevens" <williamstevens@gmail.com on behalf
of
>> >>> > wstevens@cloudops.com>
>> >>> > > > > wrote:
>> >>> > > > >
>> >>> > > > > unfortunately the template mirror conversation got
caught up
>> >>> > > > > in
>> >>> > details
>> >>> > > > and
>> >>> > > > > nobody took the lead on implementing a solution.
>> >>> > > > >
>> >>> > > > > citrix has been pinging me every couple months to
say 'dude,
>> >>> > > > > we
>> >>> need
>> >>> > to
>> >>> > > > > remove the dependency on download.citrix.com', but
i have not
>> >>> > > > > had
>> >>> > the
>> >>> > > > > cycles to get in and solve the problem.  the shutdown
of that
>> >>> > > > > is
>> >>> > > imminent
>> >>> > > > > right now, so we need to solve it asap.
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > > *Will STEVENS*
>> >>> > > > > Lead Developer
>> >>> > > > >
>> >>> > > > > <https://goo.gl/NYZ8KK>
>> >>> > > > >
>> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> >>> > paul.angus@shapeblue.com
>> >>> > > >
>> >>> > > > > wrote:
>> >>> > > > >
>> >>> > > > > > Hi Nathan,
>> >>> > > > > >
>> >>> > > > > > Ideally, if you put the template location in
(or use a
>> >>> > > > > > template
>> >>> > > defined
>> >>> > > > > > in)  test_data.py then the actual location
can be
>> >>> > > > > > overridden by
>> >>> > > anyone
>> >>> > > > > > testing.
>> >>> > > > > >
>> >>> > > > > > For Trillian, we've copied all of the templates
that people
>> >>> > > > > > have
>> >>> > > define
>> >>> > > > > to
>> >>> > > > > > a local repo and then replace the URLs in test_data.py
to
>> >>> > > > > > reduce
>> >>> > > > > bandwidth
>> >>> > > > > > use and download times.
>> >>> > > > > >
>> >>> > > > > > Ie:
>> >>> > > > > >
>> >>> > > > > >             "bootableIso":
>> >>> > > > > >                 {
>> >>> > > > > >                     "displaytext": "Test Bootable
ISO",
>> >>> > > > > >                     "name": "testISO",
>> >>> > > > > >                     "bootable": True,
>> >>> > > > > >                     "ispublic": False,
>> >>> > > > > >                     "url": "{{ marvin_images_location
>> >>> > > > > > }}/TinyCore-current.iso",
>> >>> > > > > >                     "ostype": 'Other Linux
(64-bit)',
>> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>> >>> > > > > >         },
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > I thought that we had come up with a solution
for
>> >>> > download.cloud.com
>> >>> > > ,
>> >>> > > > by
>> >>> > > > > > having a mirrorlist hosted in Community Apache
'space' with
>> >>> anyone
>> >>> > > able
>> >>> > > > > to
>> >>> > > > > > out themselves forward as a mirror.
>> >>> > > > > > But I must admit I lost track of whether anyone
made the
>> >>> requisite
>> >>> > > > > changes
>> >>> > > > > > in code....
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > Kind regards,
>> >>> > > > > >
>> >>> > > > > > Paul Angus
>> >>> > > > > >
>> >>> > > > > > paul.angus@shapeblue.com
>> >>> > > > > > www.shapeblue.com
>> >>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N
4HSUK
>> >>> > > > > > @shapeblue
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > -----Original Message-----
>> >>> > > > > > From: williamstevens@gmail.com
>> >>> > > > > > [mailto:williamstevens@gmail.com]
>> >>> > On
>> >>> > > > > > Behalf Of Will Stevens
>> >>> > > > > > Sent: 24 February 2017 16:30
>> >>> > > > > > To: dev@cloudstack.apache.org
>> >>> > > > > > Subject: Re: Modern template hosting
>> >>> > > > > >
>> >>> > > > > > this is a hard questions.  in general, we should
be setting
>> >>> > > > > > up a
>> >>> > > mirror
>> >>> > > > > on
>> >>> > > > > > some cloudstack/apache domain and then mirror
to other
>> >>> > > > > > provided
>> >>> > > > > templates.
>> >>> > > > > >
>> >>> > > > > > we MUST come up with a solution to deprecate
>> >>> > > > > > 'download.cloud.com
>> >>> ',
>> >>> > > > that
>> >>> > > > > > is going to be going away any day now.
>> >>> > > > > >
>> >>> > > > > > i don't know the right way to solve this to
be honest, but
>> >>> > > > > > if you
>> >>> > > have
>> >>> > > > > > ideas, i am willing to help.
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > >
>> >>> > > > > > *Will STEVENS*
>> >>> > > > > > Lead Developer
>> >>> > > > > >
>> >>> > > > > > <https://goo.gl/NYZ8KK>
>> >>> > > > > >
>> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson
<
>> >>> njohnson@ena.com
>> >>> > >
>> >>> > > > > wrote:
>> >>> > > > > >
>> >>> > > > > > > So not to re-open a can of worms, but
I’m in a situation
>> >>> > > > > > > where
>> >>> I
>> >>> > > need
>> >>> > > > > > > to come up with a Marvin component test
that depends on a
>> >>> > template
>> >>> > > > > > > based on a kernel that’s relatively
new, i.e., newer than
>> >>> Centos
>> >>> > > 5.3
>> >>> > > > /
>> >>> > > > > > Ubuntu 10.04 .
>> >>> > > > > > > I see openvm.eu has a suitable template
(Ubuntu 16.0.4
>> >>> > > > > > > for
>> >>> KVM),
>> >>> > > but
>> >>> > > > > > > from looking at the thread "Migrating
CloudStack content
>> >>> > > > > > > from download.cloud.com” it looks like
there is
>> >>> > > > > > > resistance to using
>> >>> > > this
>> >>> > > > at
>> >>> > > > > > > least for hosting system vm templates
over concerns of
>> >>> > neutrality.
>> >>> > > > > > > Would this be suitable for a component
test?  If not,
>> >>> > > > > > > what is a
>> >>> > > > > “blessed”
>> >>> > > > > > > template location?
>> >>> > > > > > >
>> >>> > > > > > > Thanks in advance!
>> >>> > > > > > >
>> >>> > > > > > > Nathan Johnson
>> >>> > > > > > > R&D Engineer
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville,
TN 37211
>> >>> > > > > > > General Office: 615-312-6000
>> >>> > > > > > >
>> >>> > > > > > > website | blog | support
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > >
>> >>> > > > > DISCLAIMER
>> >>> > > > > ==========
>> >>> > > > > This e-mail may contain privileged and confidential
>> >>> > > > > information
>> >>> which
>> >>> > > is
>> >>> > > > > the property of Accelerite, a Persistent Systems
business. It
>> >>> > > > > is
>> >>> > > intended
>> >>> > > > > only for the use of the individual or entity to
which it is
>> >>> > addressed.
>> >>> > > If
>> >>> > > > > you are not the intended recipient, you are not
authorized to
>> >>> > > > > read,
>> >>> > > > retain,
>> >>> > > > > copy, print, distribute or use this message. If
you have
>> >>> > > > > received
>> >>> > this
>> >>> > > > > communication in error, please notify the sender
and delete
>> >>> > > > > all
>> >>> > copies
>> >>> > > of
>> >>> > > > > this message. Accelerite, a Persistent Systems business
does
>> >>> > > > > not
>> >>> > accept
>> >>> > > > any
>> >>> > > > > liability for virus infected mails.
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >
>> >abhinandan.prateek@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>> >
>> >
>> >
>>
>> abhinandan.prateek@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message