cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: Modern template hosting
Date Mon, 27 Feb 2017 22:36:07 GMT
I think we almost need a two pronged approach.

1) Get a solution in place which will enable us to document and serve
templates for legacy systems.  I will work on this.
2) Discuss and understand how we SHOULD be handling this problem in the
future and in what release we can expect it.

I think we need to do both.  I think we should start to try to really
understand what we want to deliver in (2) going forward.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 4:53 PM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:

> My worry is exactly with system VMs templates.
>
> Currently, we indicate administrators to download them from
> http://cloudstack.apt-get.eu/systemvm/4.6/ [1]. However, the installation
> docs do not mention the expected hashes for the file that is going to be
> downloaded.
> Also, I do not know the code that downloads system VMs templates (when
> upgrading), but if the hash being checked is taken from the mirror used to
> download the file; the only thing it checks is that if the download
> finished successfully (no transmission errors). If we want to check
> integrity, check that the template we created is untampered; we need to
> host and serve the hash in a secure manner.
>
> [1]
> http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/
> management-server/index.html#prepare-the-system-vm-template
>
>
> On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <chiradeepv@gmail.com>
> wrote:
>
> > Hashes are checked (md5 IIRC) today.
> > But given the issues, I think the project should steer away from hosting
> > templates except the systemvm template.
> >
> > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Will, I think we could support different path structures. This can
> > > facilitate different deployment of mirrors based on the structure the
> > host
> > > has.
> > >
> > > Could I add something else to the discussion? Have we discussed the
> > > security impacts of setting up this mirrors approach?
> > > I mean, if any of the mirrors gets corrupted (let`s say by a hacker),
> and
> > > the templates are injected with malicious code, an attacker could
> > > potentially get un-monitored and unlimited access to a cloud
> environment.
> > >
> > > If we assume that the mirror may get malicious (it is not that I do not
> > > trust you guys, but bad things happen), we cannot host hashes there.
> > Where
> > > do you think we could store Sha512 or another hash type for these
> > > templates? Could we host in the newly proposed Github repo or maybe
> some
> > > place in the ACS website?
> > >
> > > This would have an impact on clients (needing clear documentation) and
> > our
> > > code that automatically downloads System VM templates (does it check
> > hashes
> > > when automatically installing templates today? It may require
> > > implementation changes).
> > >
> > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <wstevens@cloudops.com>
> > > wrote:
> > >
> > > > so this is what I am looking to do.  Please let me know if you have
> > > > suggestions for me or think I should be solving the problem a
> different
> > > > way.
> > > >
> > > > - We request a new Github repository from the ASF at:
> > > > 'apache/cloudstack-mirror-list'
> > > > - In this repository we track a text file in the 'gh-pages' branch
> > with a
> > > > list of valid download mirrors.
> > > > - I build a binary to be hosted by the ASF (or at least with the ASF
> > > > pointing a domain at the binary and I could potentially host it).  We
> > > will
> > > > see how they want to handle the hosting of the binary.
> > > >
> > > > The binary would expose a web server which would behave as follows:
> > > > - When the 'client' requests a download url the following flow is
> > kicked
> > > > off:
> > > > -- The mirror list is queried from github (or from a static site
> hosted
> > > on
> > > > asf, as we see fit).
> > > > -- The Lat/Lon of the 'client' is determined based on their IP.
> > > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP
> > > > lookup of the hostname.
> > > > -- The closest geographical mirror is determined, the target is
> > validated
> > > > to be available and the user is redirected.
> > > >
> > > > Some questions I have right now:
> > > > - Will every mirror have the same path structure to access the
> > equivalent
> > > > resources?
> > > > - Should we support adding a path to the mirror url to specify the
> path
> > > to
> > > > the base common path?
> > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there
> > are
> > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'.
> > > > -- If the path being requested is '
> > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in
> the
> > > > following potential paths for the mirrors:
> > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2'
> > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2'
> > > >
> > > > Does this all make sense?
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal <
> > chiradeepv@gmail.com>
> > > > wrote:
> > > >
> > > > > My bad. A few lines down, this has been added recently:
> > > > >
> > > > > this.request.setFollowRedirects(true);
> > > > >
> > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens <
> > > williamstevens@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > OK. Thanks for the heads up.
> > > > > >
> > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chiradeepv@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Sounds workable. The downloader code in the SSVM won't
follow
> > > > > redirects I
> > > > > > > think.
> > > > > > > https://github.com/apache/cloudstack/blob/
> > > > > 5511065fc20787619d9cd0444a65a3
> > > > > > > 155fc9c921/core/src/com/cloud/storage/template/
> > > > > > > HttpTemplateDownloader.java#L93
> > > > > > > https://goo.gl/dSi0r5
> > > > > > >  Might need to add
> > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy());
> > > > > > >
> > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We haven't opened a ticket yet because we don't have
a
> strategy
> > > > yet.
> > > > > > > >
> > > > > > > > What do you guys think of this:
> > > > > > > > - We setup a new github repo in the 'apache' org which
> consists
> > > of
> > > > a
> > > > > > > single
> > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > - I write a small web server, distributed as a binary,
which
> > can
> > > be
> > > > > > > hosted
> > > > > > > > by ASF Infra.  This web server will query the current
list of
> > > > mirrors
> > > > > > and
> > > > > > > > will select one and then do a 302 redirect to that
mirror.
> > > > > > > >
> > > > > > > > The act of 'choosing' a mirror could be done in a
number of
> > ways.
> > > > > > > > - If we want to define an order, then it could just
try from
> > the
> > > > top
> > > > > of
> > > > > > > the
> > > > > > > > list and work its way down.  It would curl the target
to make
> > > sure
> > > > it
> > > > > > > gets
> > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > - Or, if we want to distribute the load across the
mirrors,
> we
> > > > could
> > > > > > pick
> > > > > > > > from the list randomly.  Again, doing a curl to verify
the
> > mirror
> > > > is
> > > > > up
> > > > > > > and
> > > > > > > > then doing a redirect.
> > > > > > > > - If we want to get fancy, we could do a reverse IP
lookup
> and
> > > try
> > > > to
> > > > > > > match
> > > > > > > > the requester with their closest geographical mirror.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
<
> > > > > > chiradeepv@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What steps are needed to set up a mirror? What
does Infra
> > need
> > > to
> > > > > do?
> > > > > > > Has
> > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > >
> > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela
<
> > > > > > > > > raja.pullela@accelerite.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi will,
> > > > > > > > > >
> > > > > > > > > > I believe, we didn’t get to close ‘getting
a mirror on
> > > Apache’
> > > > > > > because
> > > > > > > > we
> > > > > > > > > > needed someone on the Apache Infra side
to close this.
> BTW,
> > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
this?) has
> > > all/most
> > > > > of
> > > > > > > the
> > > > > > > > > > content.  Once we can close on the Apache
mirror for
> > hosting
> > > > the
> > > > > > > > > content, I
> > > > > > > > > > can help assist getting the content there.
> > > > > > > > > >
> > > > > > > > > > For now, we have replicated the download.cloud.com
> content
> > > to
> > > > ‘
> > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > Also, we are working on a set of steps/procedure
to help
> > with
> > > > > this
> > > > > > > > > > change.  I will update everyone in about
a week’s time on
> > the
> > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Raja Pullela
> > > > > > > > > > Engineering Team,
> > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > >
> > > > > > > > > > On 2/24/17, 11:23 PM, "williamstevens@gmail.com
on
> behalf
> > of
> > > > > Will
> > > > > > > > > > Stevens" <williamstevens@gmail.com on
behalf of
> > > > > > > wstevens@cloudops.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > unfortunately the template mirror conversation
got caught
> > up
> > > in
> > > > > > > details
> > > > > > > > > and
> > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > >
> > > > > > > > > > citrix has been pinging me every couple
months to say
> > 'dude,
> > > we
> > > > > > need
> > > > > > > to
> > > > > > > > > > remove the dependency on download.citrix.com',
but i
> have
> > > not
> > > > > had
> > > > > > > the
> > > > > > > > > > cycles to get in and solve the problem.
 the shutdown of
> > that
> > > > is
> > > > > > > > imminent
> > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus
<
> > > > > > > paul.angus@shapeblue.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Nathan,
> > > > > > > > > > >
> > > > > > > > > > > Ideally, if you put the template location
in (or use a
> > > > template
> > > > > > > > defined
> > > > > > > > > > > in)  test_data.py then the actual location
can be
> > > overridden
> > > > by
> > > > > > > > anyone
> > > > > > > > > > > testing.
> > > > > > > > > > >
> > > > > > > > > > > For Trillian, we've copied all of the
templates that
> > people
> > > > > have
> > > > > > > > define
> > > > > > > > > > to
> > > > > > > > > > > a local repo and then replace the URLs
in test_data.py
> to
> > > > > reduce
> > > > > > > > > > bandwidth
> > > > > > > > > > > use and download times.
> > > > > > > > > > >
> > > > > > > > > > > Ie:
> > > > > > > > > > >
> > > > > > > > > > >             "bootableIso":
> > > > > > > > > > >                 {
> > > > > > > > > > >                     "displaytext":
"Test Bootable ISO",
> > > > > > > > > > >                     "name": "testISO",
> > > > > > > > > > >                     "bootable": True,
> > > > > > > > > > >                     "ispublic": False,
> > > > > > > > > > >                     "url": "{{ marvin_images_location
> > > > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > > >                     "ostype": 'Other
Linux (64-bit)',
> > > > > > > > > > >                     "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > > >         },
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I thought that we had come up with
a solution for
> > > > > > > download.cloud.com
> > > > > > > > ,
> > > > > > > > > by
> > > > > > > > > > > having a mirrorlist hosted in Community
Apache 'space'
> > with
> > > > > > anyone
> > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > > > But I must admit I lost track of whether
anyone made
> the
> > > > > > requisite
> > > > > > > > > > changes
> > > > > > > > > > > in code....
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > >
> > > > > > > > > > > Paul Angus
> > > > > > > > > > >
> > > > > > > > > > > paul.angus@shapeblue.com
> > > > > > > > > > > www.shapeblue.com
> > > > > > > > > > > 53 Chandos Place, Covent Garden, London
 WC2N 4HSUK
> > > > > > > > > > > @shapeblue
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: williamstevens@gmail.com [mailto:
> > > williamstevens@gmail.
> > > > > com]
> > > > > > > On
> > > > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > > > >
> > > > > > > > > > > this is a hard questions.  in general,
we should be
> > setting
> > > > up
> > > > > a
> > > > > > > > mirror
> > > > > > > > > > on
> > > > > > > > > > > some cloudstack/apache domain and then
mirror to other
> > > > provided
> > > > > > > > > > templates.
> > > > > > > > > > >
> > > > > > > > > > > we MUST come up with a solution to
deprecate '
> > > > > download.cloud.com
> > > > > > ',
> > > > > > > > > that
> > > > > > > > > > > is going to be going away any day now.
> > > > > > > > > > >
> > > > > > > > > > > i don't know the right way to solve
this to be honest,
> > but
> > > if
> > > > > you
> > > > > > > > have
> > > > > > > > > > > ideas, i am willing to help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan
Johnson <
> > > > > > njohnson@ena.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > So not to re-open a can of worms,
but I’m in a
> > situation
> > > > > where
> > > > > > I
> > > > > > > > need
> > > > > > > > > > > > to come up with a Marvin component
test that depends
> > on a
> > > > > > > template
> > > > > > > > > > > > based on a kernel that’s relatively
new, i.e., newer
> > than
> > > > > > Centos
> > > > > > > > 5.3
> > > > > > > > > /
> > > > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > > > I see openvm.eu has a suitable
template (Ubuntu
> 16.0.4
> > > for
> > > > > > KVM),
> > > > > > > > but
> > > > > > > > > > > > from looking at the thread "Migrating
CloudStack
> > content
> > > > from
> > > > > > > > > > > > download.cloud.com” it looks
like there is
> resistance
> > to
> > > > > using
> > > > > > > > this
> > > > > > > > > at
> > > > > > > > > > > > least for hosting system vm templates
over concerns
> of
> > > > > > > neutrality.
> > > > > > > > > > > > Would this be suitable for a component
test?  If not,
> > > what
> > > > > is a
> > > > > > > > > > “blessed”
> > > > > > > > > > > > template location?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance!
> > > > > > > > > > > >
> > > > > > > > > > > > Nathan Johnson
> > > > > > > > > > > > R&D Engineer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 618 Grassmere Park Drive, Suite
12
> > > > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > > > >
> > > > > > > > > > > > website | blog | support
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > DISCLAIMER
> > > > > > > > > > ==========
> > > > > > > > > > This e-mail may contain privileged and confidential
> > > information
> > > > > > which
> > > > > > > > is
> > > > > > > > > > the property of Accelerite, a Persistent
Systems
> business.
> > It
> > > > is
> > > > > > > > intended
> > > > > > > > > > only for the use of the individual or entity
to which it
> is
> > > > > > > addressed.
> > > > > > > > If
> > > > > > > > > > you are not the intended recipient, you
are not
> authorized
> > to
> > > > > read,
> > > > > > > > > retain,
> > > > > > > > > > copy, print, distribute or use this message.
If you have
> > > > received
> > > > > > > this
> > > > > > > > > > communication in error, please notify the
sender and
> delete
> > > all
> > > > > > > copies
> > > > > > > > of
> > > > > > > > > > this message. Accelerite, a Persistent Systems
business
> > does
> > > > not
> > > > > > > accept
> > > > > > > > > any
> > > > > > > > > > liability for virus infected mails.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message