Return-Path: X-Original-To: apmail-incubator-general-archive@www.apache.org Delivered-To: apmail-incubator-general-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 91318109E5 for ; Mon, 26 Aug 2013 23:24:23 +0000 (UTC) Received: (qmail 76732 invoked by uid 500); 26 Aug 2013 23:24:23 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 76534 invoked by uid 500); 26 Aug 2013 23:24:22 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Received: (qmail 76526 invoked by uid 99); 26 Aug 2013 23:24:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Aug 2013 23:24:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of henry.saputra@gmail.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Aug 2013 23:24:18 +0000 Received: by mail-wi0-f172.google.com with SMTP id ex4so1606190wid.5 for ; Mon, 26 Aug 2013 16:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=RdQbZJzkciweF6JuAPrhOvKZtm0vhFxzjJnZNWzDLys=; b=zrvaC+Gz1FVunwBvux+NQXEKT9uBM5cITn0DpwtE7zCR4AxNO3oSI9/eKaAjyTurv/ n8OkNTFpgOGh/xE8HfCJAWDXhDPMtsYM/iMRrnsdqzs6EuA+J7DQBFakRSuEXiWIQOd7 QzM91hf+yK/S0s9HygQY828Mq/fDffv/VhfJmn1DG2Z3FTN+RaXv+sMt7L4F76K3uuB6 sARVjUHGoqKA+vgq+psB/MoXuRLIk/aUN+8LBpQdOrd2z7V6KJgjF8J/ejPaG12zfg7I KOYoENpWcKTJxKkf+AbQjbglB0h1PziA99nJl3kzRr8d274ox9EkIegBBwt1CE4RTVMz GgAA== MIME-Version: 1.0 X-Received: by 10.180.9.69 with SMTP id x5mr9108442wia.41.1377559437483; Mon, 26 Aug 2013 16:23:57 -0700 (PDT) Received: by 10.216.161.200 with HTTP; Mon, 26 Aug 2013 16:23:57 -0700 (PDT) In-Reply-To: References: Date: Mon, 26 Aug 2013 16:23:57 -0700 Message-ID: Subject: Re: [PROPOSAL] Aurora for Incubation From: Henry Saputra To: dave@ischool.berkeley.edu Cc: "general@incubator.apache.org" Content-Type: multipart/alternative; boundary=001a11c24c16eda67204e4e20cd2 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c24c16eda67204e4e20cd2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable HI Dave, thanks for your reply. Yeah I think it would be super useful to highlight ability to use Aurora with other systems. Another question: looks like from the proposal, it does not rely on Mesos library as external dependencies at all? Like other incubator projects, would you consider not having aurora-user and aurora-issues lists at the beginning of incubation? I think we would like to have dev list as the main list to have development discussion in the early phase of incubation (other IPMCs probably would recommend the same) Thanks again, - Henry On Mon, Aug 26, 2013 at 4:13 PM, Dave Lester wro= te: > Hi Henry, > > Great question. Multiple schedulers are actively being developed for > Mesos, including Aurora by Twitter, Chronos by Airbnb ( > http://nerds.airbnb.com/introducing-chronos/), one called Marathon that I > understand will be open sourced in the future, and I've spoken with sever= al > different folks at meetups who are developing their own. There is some > overlap between these schedulers, but they also meet different use-cases = so > I still think it makes sense to create Aurora as a separate Apache projec= t > from the Mesos core. > > It's also worth noting (and perhaps this is something to highlight in the > proposal itself) that a scheduler like Aurora could potentially be used > with other systems should the community wish to pursue that development i= n > the future. > > Dave > > > On Mon, Aug 26, 2013 at 3:55 PM, Henry Saputra w= rote: > >> Hi Dave, >> >> This looks like interesting project and good contributions from Twitter >> engineering. >> >> But given that close relationship with Apache Mesos, wouldn't it be more >> effective to be delivered as extension or feature contribution to Apache >> Mesos? >> >> >> - Henry >> >> >> On Mon, Aug 26, 2013 at 3:27 PM, Dave Lester = wrote: >> >>> Hi All, >>> >>> We're pleased to share a draft ASF incubation proposal for Aurora, a >>> service scheduler used to schedule jobs onto Apache Mesos that we've >>> developed at Twitter. Aurora provides all of the primitives necessary t= o >>> quickly deploy and scale stateless and fault tolerant services in a >>> datacenter. The complete proposal can be found: >>> https://wiki.apache.org/incubator/AuroraProposal, and also pasted below= . >>> >>> In particular, we'd love to add additional mentors to the project. Your >>> feedback is appreciated. >>> >>> Dave >>> >>> =3D Abstract =3D >>> >>> Aurora is a service scheduler used to schedule jobs onto Apache Mesos. >>> >>> =3D Proposal =3D >>> >>> Aurora is a scheduler that provides all of the primitives necessary to >>> quickly deploy and scale stateless and fault tolerant services in a >>> datacenter. >>> >>> Aurora builds on top of Apache Mesos and provides common features that >>> allow any site to run large scale production applications. While the >>> project is currently used in production at Twitter, we wish to develop = a >>> community to increase contributions and see it thrive in the future. >>> >>> =3D Background =3D >>> >>> The initial development of Aurora was done at Twitter, and is planned t= o >>> be >>> open sourced. This proposal is for Aurora to join the Apache Incubator. >>> >>> =3D Rationale =3D >>> >>> While the Apache Mesos core focuses on distributing individual tasks >>> across >>> nodes in a cluster, typical services consist of dozens or hundreds of >>> replicas of tasks. As a service scheduler, Aurora provides the >>> abstraction >>> of a "job" to bundle and manage these tasks. Aurora provides many key >>> functionalities centered around a job, including: definition, the conce= pt >>> of an instance and the serverset, deployment and scheduling, health >>> checking, and introspection. It also allows cross-cutting concerns to b= e >>> handled like observability and log collection. >>> >>> =3D Current Status =3D >>> >>> =3D=3D Meritocracy =3D=3D >>> >>> By submitting this incubator proposal, we=E2=80=99re expressing our int= ent to >>> build >>> a diverse developer community around Aurora that will conduct itself >>> according to The Apache Way and use meritocratic means of accepting >>> contributions. Several members of the Aurora team overlap with Apache >>> Mesos, which successfully graduated from the Incubator and has embraced= a >>> meritocratic model of governance; we plan to follow a similar path >>> forward >>> with Aurora and believe that a synergy between both projects will make >>> this >>> even easier. >>> >>> =3D=3D Community =3D=3D >>> >>> Aurora is currently being used internally at Twitter. By open sourcing >>> the >>> project, we hope to extend our contributor base significantly and creat= e >>> a >>> vibrant community around the project. >>> >>> =3D=3D Core Developers =3D=3D >>> >>> Aurora is currently being developed by a team of seven engineers at >>> Twitter. >>> >>> =3D=3D Alignment =3D=3D >>> >>> The ASF is a natural choice to host the Aurora project, given the goal = of >>> open sourcing the project and fostering a community to grow and support >>> the >>> software. Additionally, Aurora integrates with Apache Mesos, and Apache >>> ZooKeeper for service discovery. >>> >>> We believe that inclusion within Apache will build stronger ties betwee= n >>> these projects, and create further alignment between their goals and >>> communities. >>> >>> =3D Known Risks =3D >>> >>> =3D=3D Orphaned Products =3D=3D >>> >>> The core developers plan to continue working full time on the project, >>> and >>> there is very little risk of Aurora being abandoned since it is running >>> hundreds of services as part of Twitter=E2=80=99s infrastructure. Addit= ionally, >>> members of the Mesos community beyond Twitter have expressed interest i= n >>> an >>> advanced scheduler like Aurora (see =E2=80=9CInterested Parties=E2=80= =9D section); we >>> believe that need will drive some of the community involvement necessar= y >>> for the project to incubate successfully. >>> >>> =3D=3D Inexperience with Open Source =3D=3D >>> >>> Initial Aurora committers have varying levels of experience using and >>> contributing to Open Source projects, however by working with our mento= rs >>> and the Apache community we believe we will be able to conduct ourselve= s >>> in >>> accordance with Apache Incubator guidelines. The close relationship >>> between >>> the Aurora team and Apache Mesos means there is an awareness of the >>> incubation process and a willingness to embrace The Apache Way. >>> >>> =3D=3D Homogenous Developers =3D=3D >>> >>> The initial set of committers are from a single organization, however w= e >>> expect that once approved for incubation the project will attract >>> contributors from more organizations. We have already had conversations >>> with other companies who have expressed an interest in Aurora. >>> >>> =3D=3D Reliance on Salaried Developers =3D=3D >>> >>> Initial Aurora committers are salaried developers at Twitter, however >>> shortly after open sourcing the code we plan to diversify the project= =E2=80=99s >>> core committers and contributors. >>> >>> =3D=3D Relationships with Other Apache Products =3D=3D >>> >>> Initially, Aurora has been developed as a scheduler for Apache Mesos. >>> Additionally, it relies on ZooKeeper for service discovery, allowing >>> servers to register at a location and clients to subsequently discover >>> the >>> servers. >>> >>> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D >>> >>> While we respect the reputation of the Apache brand and have no doubts >>> that >>> it will attract contributors and users, our interest is primarily to gi= ve >>> Aurora a solid home as an open source project following an established >>> development model. We have also given reasons in the Rationale and >>> Alignment sections. >>> >>> =3D Documentation =3D >>> >>> This proposal exists online as >>> http://wiki.apache.org/incubator/AuroraProposal. Project documentation >>> exists and will be released at the same time as an initial code push. >>> >>> =3D Initial Source =3D >>> >>> JIRA Aurora (AURORA) >>> >>> Source and Intellectual Property Submission Plan >>> The current codebase is available upon request. >>> >>> =3D External Dependencies =3D >>> >>> All Aurora dependencies have Apache compatible licenses, except cron4j >>> (LGPL). An alternative library, most likely Quartz (Apache Public Licen= se >>> 2.0) will be used prior to an initial code push. >>> >>> =3D Cryptography =3D >>> Not applicable. >>> >>> =3D Required Resources =3D >>> >>> =3D=3D Mailing Lists =3D=3D >>> >>> * aurora-private for private PMC discussions >>> * aurora-dev >>> * aurora-commits >>> * aurora-user >>> * aurora-issues >>> >>> =3D=3D Subversion Directory =3D=3D >>> >>> We prefer to use Git as our source control system: git:// >>> git.apache.org/aurora >>> >>> =3D=3D Issue Tracking =3D=3D >>> >>> JIRA Aurora (AURORA) >>> >>> =3D Initial Committers =3D >>> >>> * William Farner (bill at twitter dot com) >>> * Brian Wickman (wickman at twitter dot com) >>> * Kevin Sweeney (ksweeney at twitter dot com) >>> * Sathya Hariesh (sathya at twitter dot com) >>> * Jonathan Boulle (jon at twitter dot com) >>> * Maxim Khutornenko (mkhutornenko at twitter dot com) >>> * Suman Karumuri (skarumuri at twitter dot com) >>> * Dave Lester (dlester at twitter dot com) >>> >>> =3D Affiliations =3D >>> >>> Twitter >>> >>> =3D Interested Parties =3D >>> >>> * Airbnb >>> * Foursquare >>> * Mesosphere >>> >>> =3D Sponsors =3D >>> >>> =3D=3D Champion =3D=3D >>> >>> Ben Hindman >>> >>> =3D=3D Nominated Mentors =3D=3D >>> >>> * Chris Mattmann >>> * Ben Hindman >>> >>> =3D=3D Sponsoring Entity =3D=3D >>> Incubator PMC >>> >> >> > --001a11c24c16eda67204e4e20cd2--