cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiradeep Vittal <chirade...@gmail.com>
Subject Re: Modern template hosting
Date Mon, 27 Feb 2017 23:28:25 GMT
My stance is that the current workflow does a disservice to the user
community by letting them install / use outdated and insecure templates.
Now, let's assume download.cloud.com is gone forever. What do we tell ACS
users pre-4.11 as far as *built-in templates* go?
1. Direct them to update templates.sql with some new URL, but with the same
dirty old templates
2. Direct them to update templates.sql with some new URL, but with nice
templates (e.g., open.vm.eu)
3. Same as (2), but document more choices.

Now, why should things be different for 4.11 and later? Documenting the
steps to install templates offline is trivial (and can be scripted to a
large part, like cloud-install-sys-tmplt)

For pre-4.11 users, for *systemvms*, anyway we tell them to use
http://cloudstack.apt-get.eu which is not controlled by ACS.


On Mon, Feb 27, 2017 at 2:50 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> Agree with you.
> We need to support the current working flow. And then, define the first
> version that will start using the new approach.
>
> On Mon, Feb 27, 2017 at 5:36 PM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
> > I think we almost need a two pronged approach.
> >
> > 1) Get a solution in place which will enable us to document and serve
> > templates for legacy systems.  I will work on this.
> > 2) Discuss and understand how we SHOULD be handling this problem in the
> > future and in what release we can expect it.
> >
> > I think we need to do both.  I think we should start to try to really
> > understand what we want to deliver in (2) going forward.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > My worry is exactly with system VMs templates.
> > >
> > > Currently, we indicate administrators to download them from
> > > http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the
> > installation
> > > docs do not mention the expected hashes for the file that is going to
> be
> > > downloaded.
> > > Also, I do not know the code that downloads system VMs templates (when
> > > upgrading), but if the hash being checked is taken from the mirror used
> > to
> > > download the file; the only thing it checks is that if the download
> > > finished successfully (no transmission errors). If we want to check
> > > integrity, check that the template we created is untampered; we need to
> > > host and serve the hash in a secure manner.
> > >
> > > [1]
> > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > installation/en/4.9/
> > > management-server/index.html#prepare-the-system-vm-template
> > >
> > >
> > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <
> chiradeepv@gmail.com>
> > > wrote:
> > >
> > > > Hashes are checked (md5 IIRC) today.
> > > > But given the issues, I think the project should steer away from
> > hosting
> > > > templates except the systemvm template.
> > > >
> > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > Will, I think we could support different path structures. This can
> > > > > facilitate different deployment of mirrors based on the structure
> the
> > > > host
> > > > > has.
> > > > >
> > > > > Could I add something else to the discussion? Have we discussed the
> > > > > security impacts of setting up this mirrors approach?
> > > > > I mean, if any of the mirrors gets corrupted (let`s say by a
> hacker),
> > > and
> > > > > the templates are injected with malicious code, an attacker could
> > > > > potentially get un-monitored and unlimited access to a cloud
> > > environment.
> > > > >
> > > > > If we assume that the mirror may get malicious (it is not that I
do
> > not
> > > > > trust you guys, but bad things happen), we cannot host hashes
> there.
> > > > Where
> > > > > do you think we could store Sha512 or another hash type for these
> > > > > templates? Could we host in the newly proposed Github repo or maybe
> > > some
> > > > > place in the ACS website?
> > > > >
> > > > > This would have an impact on clients (needing clear documentation)
> > and
> > > > our
> > > > > code that automatically downloads System VM templates (does it
> check
> > > > hashes
> > > > > when automatically installing templates today? It may require
> > > > > implementation changes).
> > > > >
> > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > so this is what I am looking to do.  Please let me know if you
> have
> > > > > > suggestions for me or think I should be solving the problem
a
> > > different
> > > > > > way.
> > > > > >
> > > > > > - We request a new Github repository from the ASF at:
> > > > > > 'apache/cloudstack-mirror-list'
> > > > > > - In this repository we track a text file in the 'gh-pages'
> branch
> > > > with a
> > > > > > list of valid download mirrors.
> > > > > > - I build a binary to be hosted by the ASF (or at least with
the
> > ASF
> > > > > > pointing a domain at the binary and I could potentially host
it).
> > We
> > > > > will
> > > > > > see how they want to handle the hosting of the binary.
> > > > > >
> > > > > > The binary would expose a web server which would behave as
> follows:
> > > > > > - When the 'client' requests a download url the following flow
is
> > > > kicked
> > > > > > off:
> > > > > > -- The mirror list is queried from github (or from a static
site
> > > hosted
> > > > > on
> > > > > > asf, as we see fit).
> > > > > > -- The Lat/Lon of the 'client' is determined based on their
IP.
> > > > > > -- The Lat/Lon for each of the 'mirror's is determined based
on
> an
> > IP
> > > > > > lookup of the hostname.
> > > > > > -- The closest geographical mirror is determined, the target
is
> > > > validated
> > > > > > to be available and the user is redirected.
> > > > > >
> > > > > > Some questions I have right now:
> > > > > > - Will every mirror have the same path structure to access the
> > > > equivalent
> > > > > > resources?
> > > > > > - Should we support adding a path to the mirror url to specify
> the
> > > path
> > > > > to
> > > > > > the base common path?
> > > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and
> > there
> > > > are
> > > > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads
> '.
> > > > > > -- If the path being requested is '
> > > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result
> in
> > > the
> > > > > > following potential paths for the mirrors:
> > > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > > > >
> > > > > > Does this all make sense?
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > > > chiradeepv@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > My bad. A few lines down, this has been added recently:
> > > > > > >
> > > > > > > this.request.setFollowRedirects(true);
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > > > williamstevens@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > OK. Thanks for the heads up.
> > > > > > > >
> > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > chiradeepv@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sounds workable. The downloader code in the SSVM
won't
> follow
> > > > > > > redirects I
> > > > > > > > > think.
> > > > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > > > https://goo.gl/dSi0r5
> > > > > > > > >  Might need to add
> > > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
<
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We haven't opened a ticket yet because we
don't have a
> > > strategy
> > > > > > yet.
> > > > > > > > > >
> > > > > > > > > > What do you guys think of this:
> > > > > > > > > > - We setup a new github repo in the 'apache'
org which
> > > consists
> > > > > of
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > - I write a small web server, distributed
as a binary,
> > which
> > > > can
> > > > > be
> > > > > > > > > hosted
> > > > > > > > > > by ASF Infra.  This web server will query
the current
> list
> > of
> > > > > > mirrors
> > > > > > > > and
> > > > > > > > > > will select one and then do a 302 redirect
to that
> mirror.
> > > > > > > > > >
> > > > > > > > > > The act of 'choosing' a mirror could be
done in a number
> of
> > > > ways.
> > > > > > > > > > - If we want to define an order, then it
could just try
> > from
> > > > the
> > > > > > top
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > list and work its way down.  It would curl
the target to
> > make
> > > > > sure
> > > > > > it
> > > > > > > > > gets
> > > > > > > > > > a 200 and if it does, it would do a 302
redirect.
> > > > > > > > > > - Or, if we want to distribute the load
across the
> mirrors,
> > > we
> > > > > > could
> > > > > > > > pick
> > > > > > > > > > from the list randomly.  Again, doing a
curl to verify
> the
> > > > mirror
> > > > > > is
> > > > > > > up
> > > > > > > > > and
> > > > > > > > > > then doing a redirect.
> > > > > > > > > > - If we want to get fancy, we could do a
reverse IP
> lookup
> > > and
> > > > > try
> > > > > > to
> > > > > > > > > match
> > > > > > > > > > the requester with their closest geographical
mirror.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
Vittal <
> > > > > > > > chiradeepv@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > What steps are needed to set up a mirror?
What does
> Infra
> > > > need
> > > > > to
> > > > > > > do?
> > > > > > > > > Has
> > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
Pullela <
> > > > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi will,
> > > > > > > > > > > >
> > > > > > > > > > > > I believe, we didn’t get to
close ‘getting a mirror
> on
> > > > > Apache’
> > > > > > > > > because
> > > > > > > > > > we
> > > > > > > > > > > > needed someone on the Apache Infra
side to close
> this.
> > > BTW,
> > > > > > > > > > > > cloudstack-apt.get.eu (I think
Nux manages this?)
> has
> > > > > all/most
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > content.  Once we can close on
the Apache mirror for
> > > > hosting
> > > > > > the
> > > > > > > > > > > content, I
> > > > > > > > > > > > can help assist getting the content
there.
> > > > > > > > > > > >
> > > > > > > > > > > > For now, we have replicated the
download.cloud.com
> > > content
> > > > > to
> > > > > > ‘
> > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > Also, we are working on a set
of steps/procedure to
> > help
> > > > with
> > > > > > > this
> > > > > > > > > > > > change.  I will update everyone
in about a week’s
> time
> > on
> > > > the
> > > > > > > > > details.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > >
> > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on
> > > behalf
> > > > of
> > > > > > > Will
> > > > > > > > > > > > Stevens" <williamstevens@gmail.com
on behalf of
> > > > > > > > > wstevens@cloudops.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > unfortunately the template mirror
conversation got
> > caught
> > > > up
> > > > > in
> > > > > > > > > details
> > > > > > > > > > > and
> > > > > > > > > > > > nobody took the lead on implementing
a solution.
> > > > > > > > > > > >
> > > > > > > > > > > > citrix has been pinging me every
couple months to say
> > > > 'dude,
> > > > > we
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > remove the dependency on download.citrix.com',
but i
> > > have
> > > > > not
> > > > > > > had
> > > > > > > > > the
> > > > > > > > > > > > cycles to get in and solve the
problem.  the shutdown
> > of
> > > > that
> > > > > > is
> > > > > > > > > > imminent
> > > > > > > > > > > > right now, so we need to solve
it asap.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38
PM, Paul Angus <
> > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Nathan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ideally, if you put the template
location in (or
> use
> > a
> > > > > > template
> > > > > > > > > > defined
> > > > > > > > > > > > > in)  test_data.py then the
actual location can be
> > > > > overridden
> > > > > > by
> > > > > > > > > > anyone
> > > > > > > > > > > > > testing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For Trillian, we've copied
all of the templates
> that
> > > > people
> > > > > > > have
> > > > > > > > > > define
> > > > > > > > > > > > to
> > > > > > > > > > > > > a local repo and then replace
the URLs in
> > test_data.py
> > > to
> > > > > > > reduce
> > > > > > > > > > > > bandwidth
> > > > > > > > > > > > > use and download times.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ie:
> > > > > > > > > > > > >
> > > > > > > > > > > > >             "bootableIso":
> > > > > > > > > > > > >                 {
> > > > > > > > > > > > >                     "displaytext":
"Test Bootable
> > ISO",
> > > > > > > > > > > > >                     "name":
"testISO",
> > > > > > > > > > > > >                     "bootable":
True,
> > > > > > > > > > > > >                     "ispublic":
False,
> > > > > > > > > > > > >                     "url":
"{{
> marvin_images_location
> > > > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > > > >                     "ostype":
'Other Linux
> (64-bit)',
> > > > > > > > > > > > >                     "mode":
'HTTP_DOWNLOAD'
> > > > > > > > > > > > >         },
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I thought that we had come
up with a solution for
> > > > > > > > > download.cloud.com
> > > > > > > > > > ,
> > > > > > > > > > > by
> > > > > > > > > > > > > having a mirrorlist hosted
in Community Apache
> > 'space'
> > > > with
> > > > > > > > anyone
> > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > out themselves forward as
a mirror.
> > > > > > > > > > > > > But I must admit I lost track
of whether anyone
> made
> > > the
> > > > > > > > requisite
> > > > > > > > > > > > changes
> > > > > > > > > > > > > in code....
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Paul Angus
> > > > > > > > > > > > >
> > > > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > > > 53 Chandos Place, Covent
Garden, London  WC2N 4HSUK
> > > > > > > > > > > > > @shapeblue
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > From: williamstevens@gmail.com
[mailto:
> > > > > williamstevens@gmail.
> > > > > > > com]
> > > > > > > > > On
> > > > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > > > Subject: Re: Modern template
hosting
> > > > > > > > > > > > >
> > > > > > > > > > > > > this is a hard questions.
 in general, we should be
> > > > setting
> > > > > > up
> > > > > > > a
> > > > > > > > > > mirror
> > > > > > > > > > > > on
> > > > > > > > > > > > > some cloudstack/apache domain
and then mirror to
> > other
> > > > > > provided
> > > > > > > > > > > > templates.
> > > > > > > > > > > > >
> > > > > > > > > > > > > we MUST come up with a solution
to deprecate '
> > > > > > > download.cloud.com
> > > > > > > > ',
> > > > > > > > > > > that
> > > > > > > > > > > > > is going to be going away
any day now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > i don't know the right way
to solve this to be
> > honest,
> > > > but
> > > > > if
> > > > > > > you
> > > > > > > > > > have
> > > > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > >
> > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25
AM, Nathan Johnson <
> > > > > > > > njohnson@ena.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > So not to re-open a
can of worms, but I’m in a
> > > > situation
> > > > > > > where
> > > > > > > > I
> > > > > > > > > > need
> > > > > > > > > > > > > > to come up with a Marvin
component test that
> > depends
> > > > on a
> > > > > > > > > template
> > > > > > > > > > > > > > based on a kernel that’s
relatively new, i.e.,
> > newer
> > > > than
> > > > > > > > Centos
> > > > > > > > > > 5.3
> > > > > > > > > > > /
> > > > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > > > I see openvm.eu has
a suitable template (Ubuntu
> > > 16.0.4
> > > > > for
> > > > > > > > KVM),
> > > > > > > > > > but
> > > > > > > > > > > > > > from looking at the
thread "Migrating CloudStack
> > > > content
> > > > > > from
> > > > > > > > > > > > > > download.cloud.com”
it looks like there is
> > > resistance
> > > > to
> > > > > > > using
> > > > > > > > > > this
> > > > > > > > > > > at
> > > > > > > > > > > > > > least for hosting system
vm templates over
> concerns
> > > of
> > > > > > > > > neutrality.
> > > > > > > > > > > > > > Would this be suitable
for a component test?  If
> > not,
> > > > > what
> > > > > > > is a
> > > > > > > > > > > > “blessed”
> > > > > > > > > > > > > > template location?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 618 Grassmere Park Drive,
Suite 12
> > > > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > website | blog | support
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > DISCLAIMER
> > > > > > > > > > > > ==========
> > > > > > > > > > > > This e-mail may contain privileged
and confidential
> > > > > information
> > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > > the property of Accelerite, a
Persistent Systems
> > > business.
> > > > It
> > > > > > is
> > > > > > > > > > intended
> > > > > > > > > > > > only for the use of the individual
or entity to which
> > it
> > > is
> > > > > > > > > addressed.
> > > > > > > > > > If
> > > > > > > > > > > > you are not the intended recipient,
you are not
> > > authorized
> > > > to
> > > > > > > read,
> > > > > > > > > > > retain,
> > > > > > > > > > > > copy, print, distribute or use
this message. If you
> > have
> > > > > > received
> > > > > > > > > this
> > > > > > > > > > > > communication in error, please
notify the sender and
> > > delete
> > > > > all
> > > > > > > > > copies
> > > > > > > > > > of
> > > > > > > > > > > > this message. Accelerite, a Persistent
Systems
> business
> > > > does
> > > > > > not
> > > > > > > > > accept
> > > > > > > > > > > any
> > > > > > > > > > > > liability for virus infected mails.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message