Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B03DFFE86 for ; Mon, 15 Dec 2014 05:46:56 +0000 (UTC) Received: (qmail 36369 invoked by uid 500); 15 Dec 2014 05:46:56 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 36324 invoked by uid 500); 15 Dec 2014 05:46:56 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 36304 invoked by uid 99); 15 Dec 2014 05:46:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Dec 2014 05:46:56 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of connor.p.d@gmail.com designates 209.85.217.176 as permitted sender) Received: from [209.85.217.176] (HELO mail-lb0-f176.google.com) (209.85.217.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Dec 2014 05:46:30 +0000 Received: by mail-lb0-f176.google.com with SMTP id p9so8411068lbv.21 for ; Sun, 14 Dec 2014 21:45:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=/9JIY3yF4D9DqkFwrjpkXnv1i2rQw1AJy22h766UFxQ=; b=1DXE+8UmETSgF8mC19bfOhorUceNeAlBp9Mp3AlEJunZqboVHNvQANd1rEUPNXTwQD aWTBNixEPuXuOsxib2XX5535D9p3KjinOklQ5jDc/Jh8GcTWZRvkiUlkV2fDoVsKzfUd mpjSLqryTXFRWaYzeZ9eCAT1BBX0AlZHG2INRHfuTbahmBrZ3IlNnWg5onCDmL5ivYtz xiGxpLwfJ+zsxZRxd4R+VoFjVYRMInf0+B2+es3DApWDEsfNQuWBHmFpy1/p+dGwkO1G BpJDkFmZhEutAqfHwGtUbXZThhTRuqis0PsjipcRdkkL2r/m+VJir+zgzQ2gssKv/YSi 7YAg== MIME-Version: 1.0 X-Received: by 10.152.19.7 with SMTP id a7mr16405370lae.16.1418622344746; Sun, 14 Dec 2014 21:45:44 -0800 (PST) Received: by 10.152.91.35 with HTTP; Sun, 14 Dec 2014 21:45:44 -0800 (PST) In-Reply-To: References: <684552CC-4964-4602-881F-7A3240CCF8ED@mesosphere.io> <-6096449960232773984@unknownmsgid> Date: Sun, 14 Dec 2014 22:45:44 -0700 Message-ID: Subject: Re: Proposal: shared Mesos framework hosting and registry From: Connor Doyle To: "user@mesos.apache.org" , davelester@gmail.com Cc: Jing Dong , mesos , Ben Hindman , Cody Maloney , Thomas Rampelberg , Tobi Knaup Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Dave, Your rephrasing is accurate, and you do bring up some interesting points around management. Sorry for the response delay; I and the rest of the Mesosphere folks will be back in SF following our Q4 offsite by mid-week. -- Connor On Fri, Dec 12, 2014 at 12:39 PM, Dave Lester wrote: > Hey Connor, > > It would be great to hear back about this and get the ball rolling on a > shared registry. Let me know what you think! > > Thanks, > Dave > > On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester wrote: >> >> Hi Connor, >> >> Thanks for sharing the work you've been doing with Universe. Sounds >> complimentary and pretty exciting! I thought I'd take a stab at making s= ome >> clarifications. >> >> To rephrase your proposal by using a comparison, you're proposing the >> "homebrew model" of package management here in terms of how the registry= is >> maintained, correct? A registry of metadata that's hosted via a single >> GitHub repository, that community members submit pull requests to when t= hey >> want to have a release included in the registry. >> >> In this case of Homebrew, they manage "formulas"; in our case it would b= e >> package.json files (which is how NodeJS handles package info). >> >> Architecture >> The only way Universe diverges from my original thinking is that you're >> advocating decentralization of package metadata, whereas my original tho= ught >> was to have it much more centralized. There's no reason it couldn't be >> brought together through a single interface (command-line, or searchable= web >> interface) so I don't think it's incompatible technically, although I do >> think it creates some potential social/governance overhead (more below). >> >> Metadata >> The package.json file is minimal and similar to what you see in most >> package management systems. Where you're taking it a step further is a J= SON >> manifest for how to actually run the program, which is outside of the sc= ope >> of what I was hoping to accomplish but it looks awesome. >> >> Managing The Registry >> One part of your proposal that's unclear to me is when information is >> pushed to the registry repository, and how often (for each release, or j= ust >> when the package is first added). Additionally, what information is push= ed >> here -- just a pointer to the git repo? It's my current understanding th= at >> the package.json and framework.json files are both within the actual >> package's repo, not the registry. I would love to hear how you're thinki= ng >> about this. >> >> If it's just a pointer to another repo, it would make it possible for an= y >> package already vetted to become part of the registry to do their own >> updates without requiring someone to accept a PR. I think this is ideal.= If >> it's the latter, it brings up some interesting questions related to who >> would manage this and under what guidelines they would do this vetting; = we'd >> essentially get a PR for every release within the community at that poin= t. >> >> The Registry "Home" >> I feel strongly that this should be a community-driven effort, and I see >> your proposal along with my own as very compatible. Would love to work >> together, and if this works as I hope it will be a huge driver of activi= ty >> in the ecosystem. >> >> Dave >> >> On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong wrote: >>> >>> I like the idea of having a single UI for installing frameworks. One We= b >>> UI endpoint for managing setting and view state of multiple installed >>> frameworks. >>> >>> >>> On 2 Dec 2014, at 09:26, Billy Bones wrote: >>> >>> Wooo mesosphere is making such a good work pushing forward Mesos >>> technologies! >>> >>> I deeply think that Mesos should implement registry as a core feature. >>> >>> Let me explain that, Installing manually frameworks / modules / other >>> third party adds-on is quite cool when you're beginner and want to lear= n how >>> the things works, but once you're in production, you've to be fast and = react >>> quickly. >>> >>> Let said that I'm a system engineer that have customers requests like >>> "Hum... Could you had this " insert Framework/module/other name here" o= n our >>> cluster region?", I'll have to add it as soon and quick as possible on = the >>> targeted environment, I'll be OK to do it twice, but I wont make my job= a >>> request nightmare and I'll probably search for a way to provide to my >>> customers a restrictive access (WebUI) to the cluster manager and allow= them >>> to deploy those new add-on by themself. >>> >>> With this study case in mind, I'll be happy to have a registry which >>> allow a filtering mechanism to hide or show frameworks/modules/etc by s= tatus >>> (Official Apache Content / Public Content / Private Content). >>> >>> Here is my thoughts about the registry articulation: >>> >>> Mesos Registry is an integrated part of Mesos master services. >>> Mesos Registry is an endpoint available to WebUI and CLI. >>> Mesos Registry is nothing but a metadata registry. >>> Mesos Registry save its configs and metadata on a key/value store >>> (zookeeper?). >>> Mesos Registry is empty at the first launch. >>> Mesos Registry as three views: Official Apache Content | Publicly >>> Maintained Content | Private Maintained Content >>> Mesos Registry views content is: >>> >>> Official Apache Content =3D=3D Link to existing Apache add-ons hosted o= n >>> Github/Git repository + metadata (Like those proposed by Dave and Conno= r). >>> Publicly Maintained Content =3D=3D Links to existing repositories (Gith= ub / >>> Git / other) + metadata (Like those proposed by Dave and Connor). >>> Private Content =3D=3D Links to existing private (GIT/GITHUB/Other >>> repositories) + metadata (Like those proposed by Dave and Connor), this >>> repository is kinda special as it is hosted and created by the cluster >>> operators and could be a mixed content of locally maintained repository= (GIT >>> repos on a HDFS or TAR on HDFS) and public content repository cloned fr= om >>> the public content URL/Metadata. >>> >>> I don't know if I'm really clear, so if I'm not, let me know it, I'll d= o >>> some sketches :D >>> >>> >>> >>> 2014-12-02 1:53 GMT+01:00 Connor Doyle : >>>> >>>> Hi Dave, >>>> >>>> This is a timely topic, since we have been prototyping and mocking up >>>> something similar at Mesosphere. We created a new public GitHub repos= itory >>>> for it about three weeks ago called "universe" >>>> (http://github.com/mesosphere/universe). >>>> >>>> Although we have added some informal specs, it's very malleable at thi= s >>>> point. We're very much interested in making our "universe" compatible= with, >>>> or the same as, the registry you're proposing. Without delving into >>>> implementation details, some of the goals we have in mind are outlined >>>> below. >>>> >>>> Data Source: >>>> >>>> The package repository should be easily consumable by third-party >>>> command-line and other programs. There should be a condensed =E2=80= =9Cindex=E2=80=9D >>>> representation of the package repository available. >>>> >>>> Packages within the repository should be versioned. >>>> >>>> The package repository format itself should be versioned. >>>> >>>> Decentralization and Composability: >>>> >>>> The package metadata should be hosted in a public place (we like GitHu= b) >>>> so that additional packages can be added by the community by simply >>>> submitting pull requests. We have added some rudimentary commit hooks= and >>>> automated validation to protect the repo against breaking changes. >>>> >>>> It=E2=80=99s important that no single entity =E2=80=9Cowns the keys=E2= =80=9D to the universe, >>>> and that the spec and implementation remain public. It should be easy= and >>>> free for organizations to maintain a private package repository. >>>> >>>> A corollary is that it should be easy for consumers to pull from a >>>> hierarchy of upstream repositories. One setup we have in mind is that= an >>>> organization might have staging and production repositories running >>>> internally. Packages are pushed to staging where integration testing = can >>>> run before =E2=80=9Cdeployment=E2=80=9D to production. If a package i= sn=E2=80=99t in the local >>>> repository it might be looked up and installed from upstream. >>>> >>>> >>>> >>>> >>>> Repositories should be able to be proxied and cached in this way. >>>> Organizations should be able to isolate their datacenter but also >>>> selectively add external packages for experimentation. The system shou= ld be >>>> sufficiently portable and extensible to accomodate these and similar u= se >>>> cases. >>>> >>>> Meta-Framework Descriptors: >>>> >>>> Our conception of the package repository is a bit more expansive than >>>> just Mesos frameworks; it includes descriptions of how to install any = piece >>>> of server software on a Mesos cluster. Frameworks and non-frameworks = alike >>>> may be installed using some other meta-framework that=E2=80=99s respon= sible for >>>> starting all other cluster services. Likely candidates for this role = are >>>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventual= ly >>>> Kubernetes. In any case, the repository spec should not be prescripti= ve >>>> with respect to this choice. >>>> >>>> The package repository metadata should make it easy for Mesos framewor= k >>>> authors (and authors of non-Mesos-aware programs) to describe how to i= nstall >>>> their software on a Mesos cluster. To this end, our prototype package= spec >>>> allows for Meta-framework descriptor files for each package in the >>>> repository. For example for a given package we might see a `marathon.= json` >>>> file as well as a `my-app.aurora` file. >>>> >>>> An obvious concern is how to specify site-specific arguments upon >>>> installation. Here packages should describe data that must be marshal= led >>>> from the environment (e.g. by prompting a user) and combined with the = raw >>>> meta-framework descriptor to launch the app. These configuration para= meters >>>> should be agnostic of the supported meta-frameworks. More concretely,= in >>>> our prototype we describe configuration data in terms of a JSON-Schema= . >>>> >>>> CLI Integration: >>>> >>>> Part of our proposed package format is an optional descriptor for how = to >>>> fetch and install the command-line tools for interacting with the >>>> application. For now, we only have one implementation of this, which = is to >>>> fetch a python egg from PyPI. >>>> >>>> Governance: >>>> >>>> All in all, we think that making this effort more community driven is = a >>>> healthy way to proceed. Any input is very welcome. For example, if o= thers >>>> think that what we have is a good starting point we could transfer own= ership >>>> of the repository to the mesos organization on GitHub. >>>> >>>> Cheers, >>>> -- >>>> Connor Doyle >>>> http://mesosphere.com >>>> >>>> >>>> >>>> >>>> >>>> On Nov 30, 2014, at 17:32, Dave Lester wrote: >>>> >>>> As the number of Mesos frameworks grows (and now, a module system), I >>>> think it's time to create a community-maintained registry with the goa= l of >>>> making frameworks and modules easier to discover, contribute to, and >>>> install. >>>> >>>> There's already a JIRA ticket tracking this (MESOS-1759) and I've >>>> chatted with several folks (thanks in particular Victor Vieux, Tom Arn= feld, >>>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the >>>> conversation by offering a proposal on the public mailing list. >>>> >>>> I imagine two initiatives to achieve this: >>>> >>>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar wi= th >>>> how Jenkins maintains their plugins on GitHub [1], but they allow indi= vidual >>>> plugins to have their own repo within their GH org. Plugins are repos = with a >>>> specific naming convention (in their case, they've appended "-plugin" = to the >>>> repo name but this isn't the same approach we'd need to take). Having = a >>>> shared GH org creates greater visibility to what people are doing, and >>>> encourages community participation and ownership. >>>> >>>> In the case of Mesos, not all frameworks will jump at the chance to ha= ve >>>> community hosting which is fine. But particularly for smaller framewor= ks, I >>>> think this is a good option given the success Jenkins has had. We coul= d >>>> potentially use the existing /mesos github org, or create a new org >>>> /mesos-extensions or something of the like. It seems like the only pot= ential >>>> snag here is that we'll want to be explicit that these aren't Apache-b= lessed >>>> plugins and instead maintained by the larger ecosystem, but I believe = we can >>>> achieve that by offering a notice in the GH org description. >>>> >>>> 2) A registry that allows framework writers to publish new versions of >>>> their frameworks to a central repository that could be programmaticall= y >>>> accessed and rendered online. The v1 could be incredibly simple, but I= think >>>> this is a foundational piece that can grow with the project in the fut= ure. >>>> Since this is a little bit more-involved, I've created a separate Goog= le Doc >>>> [2] to begin drafting the scope for this: >>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9c= HoylylYgY/edit?usp=3Dsharing, >>>> and have intentionally punted on some of the implementation details un= til >>>> the scope is finalized and I gauge interest from users and potential >>>> collaborators. >>>> >>>> What do you think? If folks are on board I will assign the JIRA issue = to >>>> myself and get to work; I'd also welcome collaborators to help get thi= s off >>>> the ground since I think it will be a boost for the community. Feedbac= k is >>>> welcome! >>>> >>>> Thanks, >>>> Dave >>>> >>>> [1] https://github.com/jenkinsci/ >>>> [2] >>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9c= HoylylYgY/edit?usp=3Dsharing >>>> >>>> >>> >> > --=20 connor