mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Connor Doyle <con...@mesosphere.io>
Subject Re: Proposal: shared Mesos framework hosting and registry
Date Tue, 02 Dec 2014 00:53:49 GMT
Hi Dave,

This is a timely topic, since we have been prototyping and mocking up something similar at
Mesosphere.  We created a new public GitHub repository for it about three weeks ago called
"universe" (http://github.com/mesosphere/universe).

Although we have added some informal specs, it's very malleable at this point.  We're very
much interested in making our "universe" compatible with, or the same as, the registry you're
proposing.  Without delving into implementation details, some of the goals we have in mind
are outlined below.

Data Source:

The package repository should be easily consumable by third-party command-line and other programs.
 There should be a condensed “index” representation of the package repository available.

Packages within the repository should be versioned.

The package repository format itself should be versioned.

Decentralization and Composability:

The package metadata should be hosted in a public place (we like GitHub) so that additional
packages can be added by the community by simply submitting pull requests.  We have added
some rudimentary commit hooks and automated validation to protect the repo against breaking
changes.

It’s important that no single entity “owns the keys” to the universe, and that the spec
and implementation remain public.  It should be easy and free for organizations to maintain
a private package repository.

A corollary is that it should be easy for consumers to pull from a hierarchy of upstream repositories.
 One setup we have in mind is that an organization might have staging and production repositories
running internally.  Packages are pushed to staging where integration testing can run before
“deployment” to production.  If a package isn’t in the local repository it might be
looked up and installed from upstream.



Repositories should be able to be proxied and cached in this way.  Organizations should be
able to isolate their datacenter but also selectively add external packages for experimentation.
The system should be sufficiently portable and extensible to accomodate these and similar
use cases.

Meta-Framework Descriptors:

Our conception of the package repository is a bit more expansive than just Mesos frameworks;
it includes descriptions of how to install any piece of server software on a Mesos cluster.
 Frameworks and non-frameworks alike may be installed using some other meta-framework that’s
responsible for starting all other cluster services.  Likely candidates for this role are
the long-lived frameworks: Aurora, Marathon, Singularity, and eventually Kubernetes.  In any
case, the repository spec should not be prescriptive with respect to this choice.

The package repository metadata should make it easy for Mesos framework authors (and authors
of non-Mesos-aware programs) to describe how to install their software on a Mesos cluster.
 To this end, our prototype package spec allows for Meta-framework descriptor files for each
package in the repository.  For example for a given package we might see a `marathon.json`
file as well as a `my-app.aurora` file.

An obvious concern is how to specify site-specific arguments upon installation.  Here packages
should describe data that must be marshalled from the environment (e.g. by prompting a user)
and combined with the raw meta-framework descriptor to launch the app.  These configuration
parameters should be agnostic of the supported meta-frameworks.  More concretely, in our prototype
we describe configuration data in terms of a JSON-Schema.

CLI Integration:

Part of our proposed package format is an optional descriptor for how to fetch and install
the command-line tools for interacting with the application.  For now, we only have one implementation
of this, which is to fetch a python egg from PyPI.

Governance:

All in all, we think that making this effort more community driven is a healthy way to proceed.
 Any input is very welcome.  For example, if others think that what we have is a good starting
point we could transfer ownership of the repository to the mesos organization on GitHub.

Cheers,
--
Connor Doyle
http://mesosphere.com




> On Nov 30, 2014, at 17:32, Dave Lester <davelester@gmail.com> wrote:
> 
> As the number of Mesos frameworks grows (and now, a module system), I think it's time
to create a community-maintained registry with the goal of making frameworks and modules easier
to discover, contribute to, and install.
> 
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted with several
folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod Kone, Timothy St Clair, and Joe
Stein). I'd like to advance the conversation by offering a proposal on the public mailing
list.
> 
> I imagine two initiatives to achieve this:
> 
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with how Jenkins
maintains their plugins on GitHub [1], but they allow individual plugins to have their own
repo within their GH org. Plugins are repos with a specific naming convention (in their case,
they've appended "-plugin" to the repo name but this isn't the same approach we'd need to
take). Having a shared GH org creates greater visibility to what people are doing, and encourages
community participation and ownership.
> 
> In the case of Mesos, not all frameworks will jump at the chance to have community hosting
which is fine. But particularly for smaller frameworks, I think this is a good option given
the success Jenkins has had. We could potentially use the existing /mesos github org, or create
a new org /mesos-extensions or something of the like. It seems like the only potential snag
here is that we'll want to be explicit that these aren't Apache-blessed plugins and instead
maintained by the larger ecosystem, but I believe we can achieve that by offering a notice
in the GH org description.
> 
> 2) A registry that allows framework writers to publish new versions of their frameworks
to a central repository that could be programmatically accessed and rendered online. The v1
could be incredibly simple, but I think this is a foundational piece that can grow with the
project in the future. Since this is a little bit more-involved, I've created a separate Google
Doc [2] to begin drafting the scope for this: https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
and have intentionally punted on some of the implementation details until the scope is finalized
and I gauge interest from users and potential collaborators.
> 
> What do you think? If folks are on board I will assign the JIRA issue to myself and get
to work; I'd also welcome collaborators to help get this off the ground since I think it will
be a boost for the community. Feedback is welcome!
> 
> Thanks,
> Dave
> 
> [1] https://github.com/jenkinsci/
> [2] https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing


Mime
View raw message