Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
MIME-Version: 1.0
Sender: williamstevens@gmail.com
In-Reply-To: <CAEe685E5y5Uc8ujdwbP=4dtxwhO6OWZEA94Jze8kF68BGUaO=A@mail.gmail.com>
References: <17491968-94FB-4B9B-B6B9-9C6469C21B43@ena.com> <CAC2panobWV3+Tf=rXjKQpOxxDGoURfFASDXcBOr6ciZjuff96A@mail.gmail.com>
 <DB5PR07MB1334CBC991AC412624C1E5AA8C520@DB5PR07MB1334.eurprd07.prod.outlook.com>
 <CAC2panoWXcnf4oVNfNhRnJtigqCBdq=mB13giJHUrZawbdWzPg@mail.gmail.com>
 <71FCDA60-6A32-4F2F-B7D0-65FFFAA37260@persistent.co.in> <CAEe685GW-i=S4c0pq2uLtt6LA1rSF-3rNz0EWsyK6LFCfqwFUg@mail.gmail.com>
 <CAC2panqXzN1Qevgg-Vy-vpe0FT9XXvdhjA_WamD4uLQ9Z6Bo5A@mail.gmail.com>
 <CAEe685F4VZ1SJQEU1-o7ahpBSSYsSyonTh_x=Mj6=JR_cDSofA@mail.gmail.com>
 <CAC2panqS9U8Op3ODLPQmN5xMP220N_2UqsK7DxqNiO=Nq1yQ4w@mail.gmail.com> <CAEe685E5y5Uc8ujdwbP=4dtxwhO6OWZEA94Jze8kF68BGUaO=A@mail.gmail.com>
From: Will Stevens <wstevens@cloudops.com>
Date: Mon, 27 Feb 2017 15:48:39 -0500
Message-ID: <CAC2panqT+ecyWkBOgg92O5-YpAvhPU1YW7Y2F54EM-4oxJMPYA@mail.gmail.com>
Subject: Re: Modern template hosting
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary=001a113ed44c4225c405498934b5
archived-at: Mon, 27 Feb 2017 20:48:50 -0000

--001a113ed44c4225c405498934b5
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

so this is what I am looking to do.  Please let me know if you have
suggestions for me or think I should be solving the problem a different way=
.

- We request a new Github repository from the ASF at:
'apache/cloudstack-mirror-list'
- In this repository we track a text file in the 'gh-pages' branch with a
list of valid download mirrors.
- I build a binary to be hosted by the ASF (or at least with the ASF
pointing a domain at the binary and I could potentially host it).  We will
see how they want to handle the hosting of the binary.

The binary would expose a web server which would behave as follows:
- When the 'client' requests a download url the following flow is kicked
off:
-- The mirror list is queried from github (or from a static site hosted on
asf, as we see fit).
-- The Lat/Lon of the 'client' is determined based on their IP.
-- The Lat/Lon for each of the 'mirror's is determined based on an IP
lookup of the hostname.
-- The closest geographical mirror is determined, the target is validated
to be available and the user is redirected.

Some questions I have right now:
- Will every mirror have the same path structure to access the equivalent
resources?
- Should we support adding a path to the mirror url to specify the path to
the base common path?
-- Example: lets say the binary is hosted on 'dl.acs.com' and there are
three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
-- If the path being requested is '
dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
following potential paths for the mirrors:
-- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
-- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
-- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'

Does this all make sense?

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <chiradeepv@gmail.com>
wrote:

> My bad. A few lines down, this has been added recently:
>
> this.request.setFollowRedirects(true);
>
> On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <williamstevens@gmail.com>
> wrote:
>
> > OK. Thanks for the heads up.
> >
> > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chiradeepv@gmail.com>
> wrote:
> >
> > > Sounds workable. The downloader code in the SSVM won't follow
> redirects I
> > > think.
> > > https://github.com/apache/cloudstack/blob/
> 5511065fc20787619d9cd0444a65a3
> > > 155fc9c921/core/src/com/cloud/storage/template/
> > > HttpTemplateDownloader.java#L93
> > > https://goo.gl/dSi0r5
> > >  Might need to add
> > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > >
> > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <wstevens@cloudops.com>
> > > wrote:
> > >
> > > > We haven't opened a ticket yet because we don't have a strategy yet=
.
> > > >
> > > > What do you guys think of this:
> > > > - We setup a new github repo in the 'apache' org which consists of =
a
> > > single
> > > > file with a list of active/supported mirrors.
> > > > - I write a small web server, distributed as a binary, which can be
> > > hosted
> > > > by ASF Infra.  This web server will query the current list of mirro=
rs
> > and
> > > > will select one and then do a 302 redirect to that mirror.
> > > >
> > > > The act of 'choosing' a mirror could be done in a number of ways.
> > > > - If we want to define an order, then it could just try from the to=
p
> of
> > > the
> > > > list and work its way down.  It would curl the target to make sure =
it
> > > gets
> > > > a 200 and if it does, it would do a 302 redirect.
> > > > - Or, if we want to distribute the load across the mirrors, we coul=
d
> > pick
> > > > from the list randomly.  Again, doing a curl to verify the mirror i=
s
> up
> > > and
> > > > then doing a redirect.
> > > > - If we want to get fancy, we could do a reverse IP lookup and try =
to
> > > match
> > > > the requester with their closest geographical mirror.
> > > >
> > > > Thoughts?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > What steps are needed to set up a mirror? What does Infra need to
> do?
> > > Has
> > > > > anybody filed a ticket with Infra?
> > > > >
> > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > raja.pullela@accelerite.com>
> > > > > wrote:
> > > > >
> > > > > > Hi will,
> > > > > >
> > > > > > I believe, we didn=E2=80=99t get to close =E2=80=98getting a mi=
rror on Apache=E2=80=99
> > > because
> > > > we
> > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> of
> > > the
> > > > > > content.  Once we can close on the Apache mirror for hosting th=
e
> > > > > content, I
> > > > > > can help assist getting the content there.
> > > > > >
> > > > > > For now, we have replicated the download.cloud.com content to =
=E2=80=98
> > > > > > s3.download.accelerite.com=E2=80=99.
> > > > > > Also, we are working on a set of steps/procedure to help with
> this
> > > > > > change.  I will update everyone in about a week=E2=80=99s time =
on the
> > > details.
> > > > > >
> > > > > > Best,
> > > > > > Raja Pullela
> > > > > > Engineering Team,
> > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > Santa Clara, CA, 95054
> > > > > >
> > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com on behalf of
> Will
> > > > > > Stevens" <williamstevens@gmail.com on behalf of
> > > wstevens@cloudops.com>
> > > > > > wrote:
> > > > > >
> > > > > > unfortunately the template mirror conversation got caught up in
> > > details
> > > > > and
> > > > > > nobody took the lead on implementing a solution.
> > > > > >
> > > > > > citrix has been pinging me every couple months to say 'dude, we
> > need
> > > to
> > > > > > remove the dependency on download.citrix.com', but i have not
> had
> > > the
> > > > > > cycles to get in and solve the problem.  the shutdown of that i=
s
> > > > imminent
> > > > > > right now, so we need to solve it asap.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > paul.angus@shapeblue.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Nathan,
> > > > > > >
> > > > > > > Ideally, if you put the template location in (or use a templa=
te
> > > > defined
> > > > > > > in)  test_data.py then the actual location can be overridden =
by
> > > > anyone
> > > > > > > testing.
> > > > > > >
> > > > > > > For Trillian, we've copied all of the templates that people
> have
> > > > define
> > > > > > to
> > > > > > > a local repo and then replace the URLs in test_data.py to
> reduce
> > > > > > bandwidth
> > > > > > > use and download times.
> > > > > > >
> > > > > > > Ie:
> > > > > > >
> > > > > > >             "bootableIso":
> > > > > > >                 {
> > > > > > >                     "displaytext": "Test Bootable ISO",
> > > > > > >                     "name": "testISO",
> > > > > > >                     "bootable": True,
> > > > > > >                     "ispublic": False,
> > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > }}/TinyCore-current.iso",
> > > > > > >                     "ostype": 'Other Linux (64-bit)',
> > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > >         },
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I thought that we had come up with a solution for
> > > download.cloud.com
> > > > ,
> > > > > by
> > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > anyone
> > > > able
> > > > > > to
> > > > > > > out themselves forward as a mirror.
> > > > > > > But I must admit I lost track of whether anyone made the
> > requisite
> > > > > > changes
> > > > > > > in code....
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Kind regards,
> > > > > > >
> > > > > > > Paul Angus
> > > > > > >
> > > > > > > paul.angus@shapeblue.com
> > > > > > > www.shapeblue.com
> > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > @shapeblue
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: williamstevens@gmail.com [mailto:williamstevens@gmail.
> com]
> > > On
> > > > > > > Behalf Of Will Stevens
> > > > > > > Sent: 24 February 2017 16:30
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: Re: Modern template hosting
> > > > > > >
> > > > > > > this is a hard questions.  in general, we should be setting u=
p
> a
> > > > mirror
> > > > > > on
> > > > > > > some cloudstack/apache domain and then mirror to other provid=
ed
> > > > > > templates.
> > > > > > >
> > > > > > > we MUST come up with a solution to deprecate '
> download.cloud.com
> > ',
> > > > > that
> > > > > > > is going to be going away any day now.
> > > > > > >
> > > > > > > i don't know the right way to solve this to be honest, but if
> you
> > > > have
> > > > > > > ideas, i am willing to help.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > njohnson@ena.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > So not to re-open a can of worms, but I=E2=80=99m in a situ=
ation
> where
> > I
> > > > need
> > > > > > > > to come up with a Marvin component test that depends on a
> > > template
> > > > > > > > based on a kernel that=E2=80=99s relatively new, i.e., newe=
r than
> > Centos
> > > > 5.3
> > > > > /
> > > > > > > Ubuntu 10.04 .
> > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> > KVM),
> > > > but
> > > > > > > > from looking at the thread "Migrating CloudStack content fr=
om
> > > > > > > > download.cloud.com=E2=80=9D it looks like there is resistan=
ce to
> using
> > > > this
> > > > > at
> > > > > > > > least for hosting system vm templates over concerns of
> > > neutrality.
> > > > > > > > Would this be suitable for a component test?  If not, what
> is a
> > > > > > =E2=80=9Cblessed=E2=80=9D
> > > > > > > > template location?
> > > > > > > >
> > > > > > > > Thanks in advance!
> > > > > > > >
> > > > > > > > Nathan Johnson
> > > > > > > > R&D Engineer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > Nashville, TN 37211
> > > > > > > > General Office: 615-312-6000
> > > > > > > >
> > > > > > > > website | blog | support
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > DISCLAIMER
> > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > > > > This e-mail may contain privileged and confidential information
> > which
> > > > is
> > > > > > the property of Accelerite, a Persistent Systems business. It i=
s
> > > > intended
> > > > > > only for the use of the individual or entity to which it is
> > > addressed.
> > > > If
> > > > > > you are not the intended recipient, you are not authorized to
> read,
> > > > > retain,
> > > > > > copy, print, distribute or use this message. If you have receiv=
ed
> > > this
> > > > > > communication in error, please notify the sender and delete all
> > > copies
> > > > of
> > > > > > this message. Accelerite, a Persistent Systems business does no=
t
> > > accept
> > > > > any
> > > > > > liability for virus infected mails.
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a113ed44c4225c405498934b5--