cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhinandan Prateek <abhinandan.prat...@shapeblue.com>
Subject Re: Modern template hosting
Date Wed, 01 Mar 2017 12:52:33 GMT

Initial seeding is a manual step and that is not going to change in near future. A handy list
of official places from where these templates can be downloaded won’t harm this part of
the installation. Just allow anyone credible who follows democratic process to publish their
templates with apache’s blessings.


Coming to urls hard coded in db, the popular suggestion is to point them to mirrors. This
looks good but this is going to take some effort that has not materialised yet. Once this
change is made it will be going in some future release like 4.11; the problem still remains
for the releases that are already out there. In case access to download.cloud.com is dropped
at some point we will have to document the procedure such that people know how to make these
stable releases work. That is where the procedure documented here https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
should be officially adapted with a marketplace to pick up templates of choosing. Maybe we
can have a pre-setup script to make choices instead of written wiki. Note that this will be
anyway needed by the older releases.




On 01/03/17, 4:24 PM, "Paul Angus" <paul.angus@shapeblue.com> wrote:

>-1
>
>We are trying to get MORE people to use CloudStack.  And raising the barrier to entry
does nothing to help that.
>
>
>Kind regards,
>
>Paul Angus
>
>paul.angus@shapeblue.com 
>www.shapeblue.com
>53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>@shapeblue
>  
> 
>
>
>-----Original Message-----
>From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com] 
>Sent: 01 March 2017 09:39
>To: dev@cloudstack.apache.org
>Subject: Re: Modern template hosting
>
>+1
>
>After seeing this discussion running in circles several times, I think we should at least
get started with the simplest option.
>
>
>
>
>On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chiradeepv@gmail.com> wrote:
>
>>The alternative foolproof way is to simply not provide the automatic 
>>download.
>>
>>Just document the mirror list URL where the templates can be found. I 
>>don't know why people reject this option.
>>
>>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>>
>>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens 
>><williamstevens@gmail.com>
>>wrote:
>>
>>> OK. Thanks for the heads up.
>>>
>>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chiradeepv@gmail.com> wrote:
>>>
>>> > Sounds workable. The downloader code in the SSVM won't follow 
>>> > redirects I think.
>>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>>> > HttpTemplateDownloader.java#L93
>>> > https://goo.gl/dSi0r5
>>> >  Might need to add
>>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>>> >
>>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens 
>>> > <wstevens@cloudops.com>
>>> > wrote:
>>> >
>>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>>> > >
>>> > > What do you guys think of this:
>>> > > - We setup a new github repo in the 'apache' org which consists 
>>> > > of a
>>> > single
>>> > > file with a list of active/supported mirrors.
>>> > > - I write a small web server, distributed as a binary, which can 
>>> > > be
>>> > hosted
>>> > > by ASF Infra.  This web server will query the current list of 
>>> > > mirrors
>>> and
>>> > > will select one and then do a 302 redirect to that mirror.
>>> > >
>>> > > The act of 'choosing' a mirror could be done in a number of ways.
>>> > > - If we want to define an order, then it could just try from the 
>>> > > top of
>>> > the
>>> > > list and work its way down.  It would curl the target to make 
>>> > > sure it
>>> > gets
>>> > > a 200 and if it does, it would do a 302 redirect.
>>> > > - Or, if we want to distribute the load across the mirrors, we 
>>> > > could
>>> pick
>>> > > from the list randomly.  Again, doing a curl to verify the mirror 
>>> > > is up
>>> > and
>>> > > then doing a redirect.
>>> > > - If we want to get fancy, we could do a reverse IP lookup and 
>>> > > try to
>>> > match
>>> > > the requester with their closest geographical mirror.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > > *Will STEVENS*
>>> > > Lead Developer
>>> > >
>>> > > <https://goo.gl/NYZ8KK>
>>> > >
>>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>>> chiradeepv@gmail.com
>>> > >
>>> > > wrote:
>>> > >
>>> > > > What steps are needed to set up a mirror? What does Infra need
to do?
>>> > Has
>>> > > > anybody filed a ticket with Infra?
>>> > > >
>>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < 
>>> > > > raja.pullela@accelerite.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi will,
>>> > > > >
>>> > > > > I believe, we didn’t get to close ‘getting a mirror on
Apache’
>>> > because
>>> > > we
>>> > > > > needed someone on the Apache Infra side to close this. BTW,

>>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has 
>>> > > > > all/most of
>>> > the
>>> > > > > content.  Once we can close on the Apache mirror for hosting

>>> > > > > the
>>> > > > content, I
>>> > > > > can help assist getting the content there.
>>> > > > >
>>> > > > > For now, we have replicated the download.cloud.com content
to 
>>> > > > > ‘ s3.download.accelerite.com’.
>>> > > > > Also, we are working on a set of steps/procedure to help
with 
>>> > > > > this change.  I will update everyone in about a week’s
time 
>>> > > > > on the
>>> > details.
>>> > > > >
>>> > > > > Best,
>>> > > > > Raja Pullela
>>> > > > > Engineering Team,
>>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054
>>> > > > >
>>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf
of 
>>> > > > > Will Stevens" <williamstevens@gmail.com on behalf of
>>> > wstevens@cloudops.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > unfortunately the template mirror conversation got caught
up 
>>> > > > > in
>>> > details
>>> > > > and
>>> > > > > nobody took the lead on implementing a solution.
>>> > > > >
>>> > > > > citrix has been pinging me every couple months to say 'dude,

>>> > > > > we
>>> need
>>> > to
>>> > > > > remove the dependency on download.citrix.com', but i have
not 
>>> > > > > had
>>> > the
>>> > > > > cycles to get in and solve the problem.  the shutdown of
that 
>>> > > > > is
>>> > > imminent
>>> > > > > right now, so we need to solve it asap.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > *Will STEVENS*
>>> > > > > Lead Developer
>>> > > > >
>>> > > > > <https://goo.gl/NYZ8KK>
>>> > > > >
>>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>>> > paul.angus@shapeblue.com
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi Nathan,
>>> > > > > >
>>> > > > > > Ideally, if you put the template location in (or use
a 
>>> > > > > > template
>>> > > defined
>>> > > > > > in)  test_data.py then the actual location can be 
>>> > > > > > overridden by
>>> > > anyone
>>> > > > > > testing.
>>> > > > > >
>>> > > > > > For Trillian, we've copied all of the templates that
people 
>>> > > > > > have
>>> > > define
>>> > > > > to
>>> > > > > > a local repo and then replace the URLs in test_data.py
to 
>>> > > > > > reduce
>>> > > > > bandwidth
>>> > > > > > use and download times.
>>> > > > > >
>>> > > > > > Ie:
>>> > > > > >
>>> > > > > >             "bootableIso":
>>> > > > > >                 {
>>> > > > > >                     "displaytext": "Test Bootable ISO",
>>> > > > > >                     "name": "testISO",
>>> > > > > >                     "bootable": True,
>>> > > > > >                     "ispublic": False,
>>> > > > > >                     "url": "{{ marvin_images_location

>>> > > > > > }}/TinyCore-current.iso",
>>> > > > > >                     "ostype": 'Other Linux (64-bit)',
>>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
>>> > > > > >         },
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > I thought that we had come up with a solution for
>>> > download.cloud.com
>>> > > ,
>>> > > > by
>>> > > > > > having a mirrorlist hosted in Community Apache 'space'
with
>>> anyone
>>> > > able
>>> > > > > to
>>> > > > > > out themselves forward as a mirror.
>>> > > > > > But I must admit I lost track of whether anyone made
the
>>> requisite
>>> > > > > changes
>>> > > > > > in code....
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Kind regards,
>>> > > > > >
>>> > > > > > Paul Angus
>>> > > > > >
>>> > > > > > paul.angus@shapeblue.com
>>> > > > > > www.shapeblue.com
>>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK

>>> > > > > > @shapeblue
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > -----Original Message-----
>>> > > > > > From: williamstevens@gmail.com 
>>> > > > > > [mailto:williamstevens@gmail.com]
>>> > On
>>> > > > > > Behalf Of Will Stevens
>>> > > > > > Sent: 24 February 2017 16:30
>>> > > > > > To: dev@cloudstack.apache.org
>>> > > > > > Subject: Re: Modern template hosting
>>> > > > > >
>>> > > > > > this is a hard questions.  in general, we should be
setting 
>>> > > > > > up a
>>> > > mirror
>>> > > > > on
>>> > > > > > some cloudstack/apache domain and then mirror to other

>>> > > > > > provided
>>> > > > > templates.
>>> > > > > >
>>> > > > > > we MUST come up with a solution to deprecate 
>>> > > > > > 'download.cloud.com
>>> ',
>>> > > > that
>>> > > > > > is going to be going away any day now.
>>> > > > > >
>>> > > > > > i don't know the right way to solve this to be honest,
but 
>>> > > > > > if you
>>> > > have
>>> > > > > > ideas, i am willing to help.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > *Will STEVENS*
>>> > > > > > Lead Developer
>>> > > > > >
>>> > > > > > <https://goo.gl/NYZ8KK>
>>> > > > > >
>>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>>> njohnson@ena.com
>>> > >
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > So not to re-open a can of worms, but I’m in
a situation 
>>> > > > > > > where
>>> I
>>> > > need
>>> > > > > > > to come up with a Marvin component test that depends
on a
>>> > template
>>> > > > > > > based on a kernel that’s relatively new, i.e.,
newer than
>>> Centos
>>> > > 5.3
>>> > > > /
>>> > > > > > Ubuntu 10.04 .
>>> > > > > > > I see openvm.eu has a suitable template (Ubuntu
16.0.4 
>>> > > > > > > for
>>> KVM),
>>> > > but
>>> > > > > > > from looking at the thread "Migrating CloudStack
content 
>>> > > > > > > from download.cloud.com” it looks like there
is 
>>> > > > > > > resistance to using
>>> > > this
>>> > > > at
>>> > > > > > > least for hosting system vm templates over concerns
of
>>> > neutrality.
>>> > > > > > > Would this be suitable for a component test?  If
not, 
>>> > > > > > > what is a
>>> > > > > “blessed”
>>> > > > > > > template location?
>>> > > > > > >
>>> > > > > > > Thanks in advance!
>>> > > > > > >
>>> > > > > > > Nathan Johnson
>>> > > > > > > R&D Engineer
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN
37211 
>>> > > > > > > General Office: 615-312-6000
>>> > > > > > >
>>> > > > > > > website | blog | support
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > DISCLAIMER
>>> > > > > ==========
>>> > > > > This e-mail may contain privileged and confidential 
>>> > > > > information
>>> which
>>> > > is
>>> > > > > the property of Accelerite, a Persistent Systems business.
It 
>>> > > > > is
>>> > > intended
>>> > > > > only for the use of the individual or entity to which it
is
>>> > addressed.
>>> > > If
>>> > > > > you are not the intended recipient, you are not authorized
to 
>>> > > > > read,
>>> > > > retain,
>>> > > > > copy, print, distribute or use this message. If you have

>>> > > > > received
>>> > this
>>> > > > > communication in error, please notify the sender and delete

>>> > > > > all
>>> > copies
>>> > > of
>>> > > > > this message. Accelerite, a Persistent Systems business does

>>> > > > > not
>>> > accept
>>> > > > any
>>> > > > > liability for virus infected mails.
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>
>abhinandan.prateek@shapeblue.com
>www.shapeblue.com
>53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>  
> 
>

abhinandan.prateek@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Mime
View raw message