cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: Modern template hosting
Date Mon, 27 Feb 2017 21:55:50 GMT
@rafael: Yes, I think it makes sense to support sub-paths.  It will make
things a lot easier to work with.  The security question is an interesting
one.  I am not sure I would be the best person to answer that.  I would say
that we would want to have the MD5s hosted in the same repo as the mirror
list urls.

I do like the idea of running a web server on the management server for
hosting the system vms though.  It could be used to populate secondary
storage and could potentially give operators an easier (more efficient?)
workflow for introducing new templates.  That area could use some work...

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <chiradeepv@gmail.com>
wrote:

> Hashes are checked (md5 IIRC) today.
> But given the issues, I think the project should steer away from hosting
> templates except the systemvm template.
>
> On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > Will, I think we could support different path structures. This can
> > facilitate different deployment of mirrors based on the structure the
> host
> > has.
> >
> > Could I add something else to the discussion? Have we discussed the
> > security impacts of setting up this mirrors approach?
> > I mean, if any of the mirrors gets corrupted (let`s say by a hacker), and
> > the templates are injected with malicious code, an attacker could
> > potentially get un-monitored and unlimited access to a cloud environment.
> >
> > If we assume that the mirror may get malicious (it is not that I do not
> > trust you guys, but bad things happen), we cannot host hashes there.
> Where
> > do you think we could store Sha512 or another hash type for these
> > templates? Could we host in the newly proposed Github repo or maybe some
> > place in the ACS website?
> >
> > This would have an impact on clients (needing clear documentation) and
> our
> > code that automatically downloads System VM templates (does it check
> hashes
> > when automatically installing templates today? It may require
> > implementation changes).
> >
> > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <wstevens@cloudops.com>
> > wrote:
> >
> > > so this is what I am looking to do.  Please let me know if you have
> > > suggestions for me or think I should be solving the problem a different
> > > way.
> > >
> > > - We request a new Github repository from the ASF at:
> > > 'apache/cloudstack-mirror-list'
> > > - In this repository we track a text file in the 'gh-pages' branch
> with a
> > > list of valid download mirrors.
> > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > pointing a domain at the binary and I could potentially host it).  We
> > will
> > > see how they want to handle the hosting of the binary.
> > >
> > > The binary would expose a web server which would behave as follows:
> > > - When the 'client' requests a download url the following flow is
> kicked
> > > off:
> > > -- The mirror list is queried from github (or from a static site hosted
> > on
> > > asf, as we see fit).
> > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > lookup of the hostname.
> > > -- The closest geographical mirror is determined, the target is
> validated
> > > to be available and the user is redirected.
> > >
> > > Some questions I have right now:
> > > - Will every mirror have the same path structure to access the
> equivalent
> > > resources?
> > > - Should we support adding a path to the mirror url to specify the path
> > to
> > > the base common path?
> > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> are
> > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > -- If the path being requested is '
> > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in the
> > > following potential paths for the mirrors:
> > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > >
> > > Does this all make sense?
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > My bad. A few lines down, this has been added recently:
> > > >
> > > > this.request.setFollowRedirects(true);
> > > >
> > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > williamstevens@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > OK. Thanks for the heads up.
> > > > >
> > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chiradeepv@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Sounds workable. The downloader code in the SSVM won't follow
> > > > redirects I
> > > > > > think.
> > > > > > https://github.com/apache/cloudstack/blob/
> > > > 5511065fc20787619d9cd0444a65a3
> > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > HttpTemplateDownloader.java#L93
> > > > > > https://goo.gl/dSi0r5
> > > > > >  Might need to add
> > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We haven't opened a ticket yet because we don't have a
strategy
> > > yet.
> > > > > > >
> > > > > > > What do you guys think of this:
> > > > > > > - We setup a new github repo in the 'apache' org which
consists
> > of
> > > a
> > > > > > single
> > > > > > > file with a list of active/supported mirrors.
> > > > > > > - I write a small web server, distributed as a binary,
which
> can
> > be
> > > > > > hosted
> > > > > > > by ASF Infra.  This web server will query the current list
of
> > > mirrors
> > > > > and
> > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > >
> > > > > > > The act of 'choosing' a mirror could be done in a number
of
> ways.
> > > > > > > - If we want to define an order, then it could just try
from
> the
> > > top
> > > > of
> > > > > > the
> > > > > > > list and work its way down.  It would curl the target to
make
> > sure
> > > it
> > > > > > gets
> > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > - Or, if we want to distribute the load across the mirrors,
we
> > > could
> > > > > pick
> > > > > > > from the list randomly.  Again, doing a curl to verify
the
> mirror
> > > is
> > > > up
> > > > > > and
> > > > > > > then doing a redirect.
> > > > > > > - If we want to get fancy, we could do a reverse IP lookup
and
> > try
> > > to
> > > > > > match
> > > > > > > the requester with their closest geographical mirror.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > chiradeepv@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What steps are needed to set up a mirror? What does
Infra
> need
> > to
> > > > do?
> > > > > > Has
> > > > > > > > anybody filed a ticket with Infra?
> > > > > > > >
> > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi will,
> > > > > > > > >
> > > > > > > > > I believe, we didn’t get to close ‘getting
a mirror on
> > Apache’
> > > > > > because
> > > > > > > we
> > > > > > > > > needed someone on the Apache Infra side to close
this. BTW,
> > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
has
> > all/most
> > > > of
> > > > > > the
> > > > > > > > > content.  Once we can close on the Apache mirror
for
> hosting
> > > the
> > > > > > > > content, I
> > > > > > > > > can help assist getting the content there.
> > > > > > > > >
> > > > > > > > > For now, we have replicated the download.cloud.com
content
> > to
> > > ‘
> > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > Also, we are working on a set of steps/procedure
to help
> with
> > > > this
> > > > > > > > > change.  I will update everyone in about a week’s
time on
> the
> > > > > > details.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Raja Pullela
> > > > > > > > > Engineering Team,
> > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > >
> > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on behalf
> of
> > > > Will
> > > > > > > > > Stevens" <williamstevens@gmail.com on behalf
of
> > > > > > wstevens@cloudops.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > unfortunately the template mirror conversation
got caught
> up
> > in
> > > > > > details
> > > > > > > > and
> > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > >
> > > > > > > > > citrix has been pinging me every couple months
to say
> 'dude,
> > we
> > > > > need
> > > > > > to
> > > > > > > > > remove the dependency on download.citrix.com',
but i have
> > not
> > > > had
> > > > > > the
> > > > > > > > > cycles to get in and solve the problem.  the
shutdown of
> that
> > > is
> > > > > > > imminent
> > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus
<
> > > > > > paul.angus@shapeblue.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Nathan,
> > > > > > > > > >
> > > > > > > > > > Ideally, if you put the template location
in (or use a
> > > template
> > > > > > > defined
> > > > > > > > > > in)  test_data.py then the actual location
can be
> > overridden
> > > by
> > > > > > > anyone
> > > > > > > > > > testing.
> > > > > > > > > >
> > > > > > > > > > For Trillian, we've copied all of the templates
that
> people
> > > > have
> > > > > > > define
> > > > > > > > > to
> > > > > > > > > > a local repo and then replace the URLs in
test_data.py to
> > > > reduce
> > > > > > > > > bandwidth
> > > > > > > > > > use and download times.
> > > > > > > > > >
> > > > > > > > > > Ie:
> > > > > > > > > >
> > > > > > > > > >             "bootableIso":
> > > > > > > > > >                 {
> > > > > > > > > >                     "displaytext": "Test
Bootable ISO",
> > > > > > > > > >                     "name": "testISO",
> > > > > > > > > >                     "bootable": True,
> > > > > > > > > >                     "ispublic": False,
> > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > >                     "ostype": 'Other Linux
(64-bit)',
> > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > >         },
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I thought that we had come up with a solution
for
> > > > > > download.cloud.com
> > > > > > > ,
> > > > > > > > by
> > > > > > > > > > having a mirrorlist hosted in Community
Apache 'space'
> with
> > > > > anyone
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > But I must admit I lost track of whether
anyone made the
> > > > > requisite
> > > > > > > > > changes
> > > > > > > > > > in code....
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > >
> > > > > > > > > > Paul Angus
> > > > > > > > > >
> > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > www.shapeblue.com
> > > > > > > > > > 53 Chandos Place, Covent Garden, London
 WC2N 4HSUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > williamstevens@gmail.
> > > > com]
> > > > > > On
> > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > >
> > > > > > > > > > this is a hard questions.  in general, we
should be
> setting
> > > up
> > > > a
> > > > > > > mirror
> > > > > > > > > on
> > > > > > > > > > some cloudstack/apache domain and then mirror
to other
> > > provided
> > > > > > > > > templates.
> > > > > > > > > >
> > > > > > > > > > we MUST come up with a solution to deprecate
'
> > > > download.cloud.com
> > > > > ',
> > > > > > > > that
> > > > > > > > > > is going to be going away any day now.
> > > > > > > > > >
> > > > > > > > > > i don't know the right way to solve this
to be honest,
> but
> > if
> > > > you
> > > > > > > have
> > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
Johnson <
> > > > > njohnson@ena.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > So not to re-open a can of worms, but
I’m in a
> situation
> > > > where
> > > > > I
> > > > > > > need
> > > > > > > > > > > to come up with a Marvin component
test that depends
> on a
> > > > > > template
> > > > > > > > > > > based on a kernel that’s relatively
new, i.e., newer
> than
> > > > > Centos
> > > > > > > 5.3
> > > > > > > > /
> > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > I see openvm.eu has a suitable template
(Ubuntu 16.0.4
> > for
> > > > > KVM),
> > > > > > > but
> > > > > > > > > > > from looking at the thread "Migrating
CloudStack
> content
> > > from
> > > > > > > > > > > download.cloud.com” it looks like
there is resistance
> to
> > > > using
> > > > > > > this
> > > > > > > > at
> > > > > > > > > > > least for hosting system vm templates
over concerns of
> > > > > > neutrality.
> > > > > > > > > > > Would this be suitable for a component
test?  If not,
> > what
> > > > is a
> > > > > > > > > “blessed”
> > > > > > > > > > > template location?
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance!
> > > > > > > > > > >
> > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > R&D Engineer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > >
> > > > > > > > > > > website | blog | support
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > DISCLAIMER
> > > > > > > > > ==========
> > > > > > > > > This e-mail may contain privileged and confidential
> > information
> > > > > which
> > > > > > > is
> > > > > > > > > the property of Accelerite, a Persistent Systems
business.
> It
> > > is
> > > > > > > intended
> > > > > > > > > only for the use of the individual or entity
to which it is
> > > > > > addressed.
> > > > > > > If
> > > > > > > > > you are not the intended recipient, you are not
authorized
> to
> > > > read,
> > > > > > > > retain,
> > > > > > > > > copy, print, distribute or use this message.
If you have
> > > received
> > > > > > this
> > > > > > > > > communication in error, please notify the sender
and delete
> > all
> > > > > > copies
> > > > > > > of
> > > > > > > > > this message. Accelerite, a Persistent Systems
business
> does
> > > not
> > > > > > accept
> > > > > > > > any
> > > > > > > > > liability for virus infected mails.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message