cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@shapeblue.com>
Subject Re: Modern template hosting
Date Fri, 03 Mar 2017 06:24:19 GMT
Nice little thing Will,

One question: if I read the code correctly it ‘go’es and tries all mirrors at once and
whichever responses first is redirected to the client. This might well be the same every time.
This might be the one closest to the mirror302 and not to the client. Is that correct or did
I miss a weighing algorithm hidden in there?

Good coding,

On 03/03/17 00:23, "Will Stevens" <swill@apache.org> wrote:

    Hey All,
    Please review this repo and tell me what you think:
    https://github.com/swill/mirror302
    
    Let me know if you have questions or you would like me to make changes
    before I bring this topic up with ASF Infra and open a ticket to get this
    implemented.
    
    Cheers,
    
    *Will Stevens*
    
    
    
    On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstevens@cloudops.com> wrote:
    
    > I am building a short term solution right now.  I hopefully will have the
    > building blocks in place by the weekend so we can start working with Infra
    > to get it in place.
    >
    > We will have a web server which we can point something like
    > downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
    > repo which tracks a mirror list and exposes a static site to be able to
    > query the mirror list.
    >
    > The web server will receive a request and will do a 302 redirect to the
    > appropriate resource on one of the mirrors.
    >
    > This gets us started.  Legacy environments will have to do a DB change to
    > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
    > whatever), but otherwise it should be pretty simple.
    >
    > We can improve how we deliver templates going forward, but this obviously
    > requires some discussion still.  I will try to buy us some time with an
    > implementation which solves our problems today...
    >
    > *Will STEVENS*
    > Lead Developer
    >
    > <https://goo.gl/NYZ8KK>
    >
    > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
    > abhinandan.prateek@shapeblue.com> wrote:
    >
    >>
    >> Initial seeding is a manual step and that is not going to change in near
    >> future. A handy list of official places from where these templates can be
    >> downloaded won’t harm this part of the installation. Just allow anyone
    >> credible who follows democratic process to publish their templates with
    >> apache’s blessings.
    >>
    >>
    >> Coming to urls hard coded in db, the popular suggestion is to point them
    >> to mirrors. This looks good but this is going to take some effort that has
    >> not materialised yet. Once this change is made it will be going in some
    >> future release like 4.11; the problem still remains for the releases that
    >> are already out there. In case access to download.cloud.com is dropped
    >> at some point we will have to document the procedure such that people know
    >> how to make these stable releases work. That is where the procedure
    >> documented here https://shankerbalan.net/blog/
    >> seed-cloudstack-templates-offline/ should be officially adapted with a
    >> marketplace to pick up templates of choosing. Maybe we can have a pre-setup
    >> script to make choices instead of written wiki. Note that this will be
    >> anyway needed by the older releases.
    >>
    >>
    >>
    >>
    >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.angus@shapeblue.com> wrote:
    >>
    >> >-1
    >> >
    >> >We are trying to get MORE people to use CloudStack.  And raising the
    >> barrier to entry does nothing to help that.
    >> >
    >> >
    >> >Kind regards,
    >> >
    >> >Paul Angus
    >> >
    >> >paul.angus@shapeblue.com
    >> >www.shapeblue.com
    >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    >> >@shapeblue
    >> >
    >> >
    >> >
    >> >
    >> >-----Original Message-----
    >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@shapeblue.com]
    >> >Sent: 01 March 2017 09:39
    >> >To: dev@cloudstack.apache.org
    >> >Subject: Re: Modern template hosting
    >> >
    >> >+1
    >> >
    >> >After seeing this discussion running in circles several times, I think
    >> we should at least get started with the simplest option.
    >> >
    >> >
    >> >
    >> >
    >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chiradeepv@gmail.com> wrote:
    >> >
    >> >>The alternative foolproof way is to simply not provide the automatic
    >> >>download.
    >> >>
    >> >>Just document the mirror list URL where the templates can be found. I
    >> >>don't know why people reject this option.
    >> >>
    >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
    >> >>
    >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
    >> >><williamstevens@gmail.com>
    >> >>wrote:
    >> >>
    >> >>> OK. Thanks for the heads up.
    >> >>>
    >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chiradeepv@gmail.com>
    >> wrote:
    >> >>>
    >> >>> > Sounds workable. The downloader code in the SSVM won't follow
    >> >>> > redirects I think.
    >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
    >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
    >> >>> > HttpTemplateDownloader.java#L93
    >> >>> > https://goo.gl/dSi0r5
    >> >>> >  Might need to add
    >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
    >> >>> >
    >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
    >> >>> > <wstevens@cloudops.com>
    >> >>> > wrote:
    >> >>> >
    >> >>> > > We haven't opened a ticket yet because we don't have a
strategy
    >> yet.
    >> >>> > >
    >> >>> > > What do you guys think of this:
    >> >>> > > - We setup a new github repo in the 'apache' org which
consists
    >> >>> > > of a
    >> >>> > single
    >> >>> > > file with a list of active/supported mirrors.
    >> >>> > > - I write a small web server, distributed as a binary,
which can
    >> >>> > > be
    >> >>> > hosted
    >> >>> > > by ASF Infra.  This web server will query the current
list of
    >> >>> > > mirrors
    >> >>> and
    >> >>> > > will select one and then do a 302 redirect to that mirror.
    >> >>> > >
    >> >>> > > The act of 'choosing' a mirror could be done in a number
of ways.
    >> >>> > > - If we want to define an order, then it could just try
from the
    >> >>> > > top of
    >> >>> > the
    >> >>> > > list and work its way down.  It would curl the target
to make
    >> >>> > > sure it
    >> >>> > gets
    >> >>> > > a 200 and if it does, it would do a 302 redirect.
    >> >>> > > - Or, if we want to distribute the load across the mirrors,
we
    >> >>> > > could
    >> >>> pick
    >> >>> > > from the list randomly.  Again, doing a curl to verify
the mirror
    >> >>> > > is up
    >> >>> > and
    >> >>> > > then doing a redirect.
    >> >>> > > - If we want to get fancy, we could do a reverse IP lookup
and
    >> >>> > > try to
    >> >>> > match
    >> >>> > > the requester with their closest geographical mirror.
    >> >>> > >
    >> >>> > > Thoughts?
    >> >>> > >
    >> >>> > > *Will STEVENS*
    >> >>> > > Lead Developer
    >> >>> > >
    >> >>> > > <https://goo.gl/NYZ8KK>
    >> >>> > >
    >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
    >> >>> chiradeepv@gmail.com
    >> >>> > >
    >> >>> > > wrote:
    >> >>> > >
    >> >>> > > > What steps are needed to set up a mirror? What does
Infra need
    >> to do?
    >> >>> > Has
    >> >>> > > > anybody filed a ticket with Infra?
    >> >>> > > >
    >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
    >> >>> > > > raja.pullela@accelerite.com>
    >> >>> > > > wrote:
    >> >>> > > >
    >> >>> > > > > Hi will,
    >> >>> > > > >
    >> >>> > > > > I believe, we didn’t get to close ‘getting
a mirror on Apache’
    >> >>> > because
    >> >>> > > we
    >> >>> > > > > needed someone on the Apache Infra side to close
this. BTW,
    >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?)
has
    >> >>> > > > > all/most of
    >> >>> > the
    >> >>> > > > > content.  Once we can close on the Apache mirror
for hosting
    >> >>> > > > > the
    >> >>> > > > content, I
    >> >>> > > > > can help assist getting the content there.
    >> >>> > > > >
    >> >>> > > > > For now, we have replicated the download.cloud.com
content to
    >> >>> > > > > ‘ s3.download.accelerite.com’.
    >> >>> > > > > Also, we are working on a set of steps/procedure
to help with
    >> >>> > > > > this change.  I will update everyone in about
a week’s time
    >> >>> > > > > on the
    >> >>> > details.
    >> >>> > > > >
    >> >>> > > > > Best,
    >> >>> > > > > Raja Pullela
    >> >>> > > > > Engineering Team,
    >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara,
CA, 95054
    >> >>> > > > >
    >> >>> > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on behalf of
    >> >>> > > > > Will Stevens" <williamstevens@gmail.com on
behalf of
    >> >>> > wstevens@cloudops.com>
    >> >>> > > > > wrote:
    >> >>> > > > >
    >> >>> > > > > unfortunately the template mirror conversation
got caught up
    >> >>> > > > > in
    >> >>> > details
    >> >>> > > > and
    >> >>> > > > > nobody took the lead on implementing a solution.
    >> >>> > > > >
    >> >>> > > > > citrix has been pinging me every couple months
to say 'dude,
    >> >>> > > > > we
    >> >>> need
    >> >>> > to
    >> >>> > > > > remove the dependency on download.citrix.com',
but i have not
    >> >>> > > > > had
    >> >>> > the
    >> >>> > > > > cycles to get in and solve the problem.  the
shutdown of that
    >> >>> > > > > is
    >> >>> > > imminent
    >> >>> > > > > right now, so we need to solve it asap.
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > > *Will STEVENS*
    >> >>> > > > > Lead Developer
    >> >>> > > > >
    >> >>> > > > > <https://goo.gl/NYZ8KK>
    >> >>> > > > >
    >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus
<
    >> >>> > paul.angus@shapeblue.com
    >> >>> > > >
    >> >>> > > > > wrote:
    >> >>> > > > >
    >> >>> > > > > > Hi Nathan,
    >> >>> > > > > >
    >> >>> > > > > > Ideally, if you put the template location
in (or use a
    >> >>> > > > > > template
    >> >>> > > defined
    >> >>> > > > > > in)  test_data.py then the actual location
can be
    >> >>> > > > > > overridden by
    >> >>> > > anyone
    >> >>> > > > > > testing.
    >> >>> > > > > >
    >> >>> > > > > > For Trillian, we've copied all of the templates
that people
    >> >>> > > > > > have
    >> >>> > > define
    >> >>> > > > > to
    >> >>> > > > > > a local repo and then replace the URLs
in test_data.py to
    >> >>> > > > > > reduce
    >> >>> > > > > bandwidth
    >> >>> > > > > > use and download times.
    >> >>> > > > > >
    >> >>> > > > > > Ie:
    >> >>> > > > > >
    >> >>> > > > > >             "bootableIso":
    >> >>> > > > > >                 {
    >> >>> > > > > >                     "displaytext": "Test
Bootable ISO",
    >> >>> > > > > >                     "name": "testISO",
    >> >>> > > > > >                     "bootable": True,
    >> >>> > > > > >                     "ispublic": False,
    >> >>> > > > > >                     "url": "{{ marvin_images_location
    >> >>> > > > > > }}/TinyCore-current.iso",
    >> >>> > > > > >                     "ostype": 'Other Linux
(64-bit)',
    >> >>> > > > > >                     "mode": 'HTTP_DOWNLOAD'
    >> >>> > > > > >         },
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > I thought that we had come up with a solution
for
    >> >>> > download.cloud.com
    >> >>> > > ,
    >> >>> > > > by
    >> >>> > > > > > having a mirrorlist hosted in Community
Apache 'space' with
    >> >>> anyone
    >> >>> > > able
    >> >>> > > > > to
    >> >>> > > > > > out themselves forward as a mirror.
    >> >>> > > > > > But I must admit I lost track of whether
anyone made the
    >> >>> requisite
    >> >>> > > > > changes
    >> >>> > > > > > in code....
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > Kind regards,
    >> >>> > > > > >
    >> >>> > > > > > Paul Angus
    >> >>> > > > > >
    >> >>> > > > > > paul.angus@shapeblue.com
    >> >>> > > > > > www.shapeblue.com
    >> >>> > > > > > 53 Chandos Place, Covent Garden, London
 WC2N 4HSUK
    >> >>> > > > > > @shapeblue
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > -----Original Message-----
    >> >>> > > > > > From: williamstevens@gmail.com
    >> >>> > > > > > [mailto:williamstevens@gmail.com]
    >> >>> > On
    >> >>> > > > > > Behalf Of Will Stevens
    >> >>> > > > > > Sent: 24 February 2017 16:30
    >> >>> > > > > > To: dev@cloudstack.apache.org
    >> >>> > > > > > Subject: Re: Modern template hosting
    >> >>> > > > > >
    >> >>> > > > > > this is a hard questions.  in general,
we should be setting
    >> >>> > > > > > up a
    >> >>> > > mirror
    >> >>> > > > > on
    >> >>> > > > > > some cloudstack/apache domain and then
mirror to other
    >> >>> > > > > > provided
    >> >>> > > > > templates.
    >> >>> > > > > >
    >> >>> > > > > > we MUST come up with a solution to deprecate
    >> >>> > > > > > 'download.cloud.com
    >> >>> ',
    >> >>> > > > that
    >> >>> > > > > > is going to be going away any day now.
    >> >>> > > > > >
    >> >>> > > > > > i don't know the right way to solve this
to be honest, but
    >> >>> > > > > > if you
    >> >>> > > have
    >> >>> > > > > > ideas, i am willing to help.
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > >
    >> >>> > > > > > *Will STEVENS*
    >> >>> > > > > > Lead Developer
    >> >>> > > > > >
    >> >>> > > > > > <https://goo.gl/NYZ8KK>
    >> >>> > > > > >
    >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
Johnson <
    >> >>> njohnson@ena.com
    >> >>> > >
    >> >>> > > > > wrote:
    >> >>> > > > > >
    >> >>> > > > > > > So not to re-open a can of worms,
but I’m in a situation
    >> >>> > > > > > > where
    >> >>> I
    >> >>> > > need
    >> >>> > > > > > > to come up with a Marvin component
test that depends on a
    >> >>> > template
    >> >>> > > > > > > based on a kernel that’s relatively
new, i.e., newer than
    >> >>> Centos
    >> >>> > > 5.3
    >> >>> > > > /
    >> >>> > > > > > Ubuntu 10.04 .
    >> >>> > > > > > > I see openvm.eu has a suitable template
(Ubuntu 16.0.4
    >> >>> > > > > > > for
    >> >>> KVM),
    >> >>> > > but
    >> >>> > > > > > > from looking at the thread "Migrating
CloudStack content
    >> >>> > > > > > > from download.cloud.com” it looks
like there is
    >> >>> > > > > > > resistance to using
    >> >>> > > this
    >> >>> > > > at
    >> >>> > > > > > > least for hosting system vm templates
over concerns of
    >> >>> > neutrality.
    >> >>> > > > > > > Would this be suitable for a component
test?  If not,
    >> >>> > > > > > > what is a
    >> >>> > > > > “blessed”
    >> >>> > > > > > > template location?
    >> >>> > > > > > >
    >> >>> > > > > > > Thanks in advance!
    >> >>> > > > > > >
    >> >>> > > > > > > Nathan Johnson
    >> >>> > > > > > > R&D Engineer
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12
Nashville, TN 37211
    >> >>> > > > > > > General Office: 615-312-6000
    >> >>> > > > > > >
    >> >>> > > > > > > website | blog | support
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > > >
    >> >>> > > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > >
    >> >>> > > > > DISCLAIMER
    >> >>> > > > > ==========
    >> >>> > > > > This e-mail may contain privileged and confidential
    >> >>> > > > > information
    >> >>> which
    >> >>> > > is
    >> >>> > > > > the property of Accelerite, a Persistent Systems
business. It
    >> >>> > > > > is
    >> >>> > > intended
    >> >>> > > > > only for the use of the individual or entity
to which it is
    >> >>> > addressed.
    >> >>> > > If
    >> >>> > > > > you are not the intended recipient, you are
not authorized to
    >> >>> > > > > read,
    >> >>> > > > retain,
    >> >>> > > > > copy, print, distribute or use this message.
If you have
    >> >>> > > > > received
    >> >>> > this
    >> >>> > > > > communication in error, please notify the sender
and delete
    >> >>> > > > > all
    >> >>> > copies
    >> >>> > > of
    >> >>> > > > > this message. Accelerite, a Persistent Systems
business does
    >> >>> > > > > not
    >> >>> > accept
    >> >>> > > > any
    >> >>> > > > > liability for virus infected mails.
    >> >>> > > > >
    >> >>> > > >
    >> >>> > >
    >> >>> >
    >> >>>
    >> >
    >> >abhinandan.prateek@shapeblue.com
    >> >www.shapeblue.com
    >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
    >> >
    >> >
    >> >
    >>
    >> abhinandan.prateek@shapeblue.com
    >> www.shapeblue.com
    >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    >> @shapeblue
    >>
    >>
    >>
    >>
    >
    


daan.hoogland@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Mime
View raw message