Return-Path: Delivered-To: apmail-incubator-general-archive@www.apache.org Received: (qmail 28967 invoked from network); 21 Dec 2010 07:37:16 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 Dec 2010 07:37:16 -0000 Received: (qmail 20763 invoked by uid 500); 21 Dec 2010 07:37:15 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 20104 invoked by uid 500); 21 Dec 2010 07:37:12 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Received: (qmail 20096 invoked by uid 99); 21 Dec 2010 07:37:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 07:37:11 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 21 Dec 2010 07:37:07 +0000 Received: (qmail 28738 invoked by uid 99); 21 Dec 2010 07:36:45 -0000 Received: from localhost.apache.org (HELO mail-iw0-f175.google.com) (127.0.0.1) (smtp-auth username edwardyoon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 07:36:45 +0000 Received: by iwn8 with SMTP id 8so4091756iwn.6 for ; Mon, 20 Dec 2010 23:36:45 -0800 (PST) MIME-Version: 1.0 Received: by 10.231.15.204 with SMTP id l12mr4992921iba.148.1292917005066; Mon, 20 Dec 2010 23:36:45 -0800 (PST) Received: by 10.231.11.201 with HTTP; Mon, 20 Dec 2010 23:36:45 -0800 (PST) In-Reply-To: References: <4D0D2857.8000100@apache.org> Date: Tue, 21 Dec 2010 16:36:45 +0900 Message-ID: Subject: Re: [VOTE] Mesos to enter the incubator From: "Edward J. Yoon" To: general@incubator.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org +1 non-binding On Tue, Dec 21, 2010 at 1:02 AM, Brian McCallister wrote= : > +1 > > On Sat, Dec 18, 2010 at 2:32 PM, Matei Zaharia wrote: >> We've finalized our proposal for Mesos >> (http://wiki.apache.org/incubator/MesosProposal) and we'd now like to pu= t it >> up for vote. >> >> I'll tally the results after five days, on December 23rd. >> >> Thanks, >> >> Matei >> >> >> >> >> --------------------------------------------------------------------- >> MESOS PROPOSAL >> --------------------------------------------------------------------- >> >> >> =3D Abstract =3D >> >> Mesos is a cluster manager that provides resource sharing and >> isolation across cluster applications. >> >> >> >> =3D Proposal =3D >> >> Mesos is system for sharing resources between cluster applications such >> as Hadoop MapReduce, HBase, MPI, and web applications. >> It is motivated by three use cases. First, organizations that use >> several of these applications can use Mesos to share nodes between them, >> increasing utilization and simplifying management. Second, inspired by >> MapReduce, a wide array of new cluster programming frameworks are being >> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and >> Caffeine. Mesos provides a common interface for such frameworks to share >> resources, allowing organizations to use multiple frameworks in the same >> cluster. Third, Mesos allows users of a framework such as Hadoop to have >> multiple instances of the framework on the same cluster, facilitating >> workload isolation and incremental deployment of upgrades. >> >> >> >> =3D Background =3D >> >> Mesos was inspired by operational issues experienced in large Apache Had= oop >> deployments as well as a desire to provide a management system for a >> wider range of cluster applications. The Apache Hadoop community has lon= g >> realized that the current model of having one instance of MapReduce >> control a whole cluster leads to problems with isolation (one job may >> cause the master to crash, killing all the other jobs), scalability, >> and software upgrades (an upgrade must be deployed on the whole cluster)= . >> Statically partitioning resources into multiple fixed-size MapReduce >> clusters >> is unattractive because it lowers both utilization and data locality. >> The community has discussed a two-level scheduling model where a simple, >> robust low-level layer enables multiple applications to launch tasks >> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a >> layer, >> with the additional goal of supporting non-Hadoop applications as well. >> >> Mesos started as a research project at UC Berkeley, but is now being >> tested at several companies (including Twitter and Facebook), and has >> attracted >> interest from other industry users and researchers as well. We are >> therefore proposing to place Mesos in the Apache incubator and build an >> open source community around it. >> >> >> >> =3D Rationale =3D >> >> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine) >> already exist in the scientific computing community, they are not well >> suited for today's data center environment. >> These schedulers generally give jobs coarse-grained static allocations o= f >> the cluster (e.g. X nodes for the full duration of the job). >> This is problematic because many cluster applications are elastic >> (can scale up and down), so utilization is not optimal under static >> partitioning, and because data-intensive applications such as MapReduce >> need to run a few tasks on every node of the cluster to read data locall= y. >> To address these challenges, Mesos is designed around two principles: >> >> =C2=A0* Fine-grained sharing: Mesos allocates resources at the level of = "tasks" >> =C2=A0 within a job, allowing applications to scale up and down over tim= e and >> =C2=A0 to take turns accessing data on cluster nodes. >> =C2=A0* Application-controlled scheduling: Applications control which no= des >> =C2=A0 their tasks run on, allowing them to achieve placement goals such= as >> =C2=A0 data locality. >> >> In addition to these principles, Mesos is designed to be simple, scalabl= e >> and robust, becuase a cluster manager must be highly available to suppor= t >> applications and should not become a bottleneck. Application-controlled >> scheduling already simplifies our design by pushing much of the complex >> logic of tracking job state to applications. In addition, Mesos employs = an >> optimized C++ message-passing library to achieve scalability and support= s >> master failover using Apache ZooKeeper. >> >> Mesos already supports running Hadoop and MPI. We plan to add support >> for other systems as requested (and contributed) by the community. >> >> >> >> =3D Current Status =3D >> >> =3D=3D Meritocracy =3D=3D >> >> Our intent with this incubator proposal is to start building a diverse >> developer community around Mesos following the Apache meritocracy model. >> We have wanted to make the project open source and encourage contributor= s >> from multiple organizations from the start. We plan to provide plenty >> of support to new developers and to quickly recruit those who make solid >> contributions to committer status. >> >> =3D=3D Community =3D=3D >> >> Mesos is currently being used by developers at Twitter and researchers i= n >> computer science and civil engineering at Berkeley. We hope to extend th= e >> user >> and developer base further in the future. The current developers and use= rs >> are all interested in building a solid open source community around Meso= s. >> >> To work towards an open source community, we have been using the GitHub >> issue >> tracker and mailing lists at Berkeley for development discussions within= our >> group for several months now. >> >> =3D=3D Core Developers =3D=3D >> >> Mesos was started by three graduate students at UC Berkeley (Benjamin >> Hindman, >> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc fro= m >> the Swedish Institute of Computer Science (Ali Ghodsi). Although started= as >> a research project, Mesos was always intended to solve operational issue= s >> with large clusters and to become an open-source project, building on ou= r >> successful experience doing research that has been incorporated into Apa= che >> Hadoop >> (several scheduling algorithms). >> >> =3D=3D Alignment =3D=3D >> >> The ASF is a natural host for Mesos given that it is already the home of >> Hadoop, HBase, Cassandra, and other emerging cloud software projects. >> Mesos was designed to support Hadoop from the beginning in order to solv= e >> operational challenges in Hadoop clusters, and it aims to support a wide >> range >> of applications beyond Hadoop as well. Mesos complements the existing Ap= ache >> cloud computing projects by providing a unified way to manage these syst= ems >> and to share resources and data between them. >> >> >> >> =3D Known Risks =3D >> >> =3D=3D Orphaned Products =3D=3D >> >> With the current core developers of Mesos being graduate students, there >> is a risk that these developers will eventually move on to other project= s. >> However, because of the broad scope of Mesos, we all plan to continue >> working >> on projects related to it in the next several years. We are also activel= y >> working with developers at other organizations, such as Twitter, who are >> good candidates to become contributors. >> >> =3D=3D Inexperience with Open Source =3D=3D >> >> All of the core developers are active users and followers of open source= . >> Matei Zaharia is a Hadoop committer and has experience with the Apache >> infrastructure and development process. Andy Konwinski has contributed >> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi >> has released open source software as part of his PhD work that was adopt= ed >> by a Swedish company. >> >> =3D=3D Homogeneous Developers =3D=3D >> >> The current core developers are all researchers (graduate students and a >> young professor). However, we hope to establish a developer community >> that includes contributors from several corporations, and we are already >> working towards this with Twitter and Facebook. >> >> =3D=3D Reliance on Salaried Developers =3D=3D >> >> Given that the project started in an academic research environment, the >> core developers are all interested in it primarily for its own sake rath= er >> than for the sake of employment. We all intend to continue working on Me= sos >> as volunteers. >> >> =3D=3D Relationships with Other Apache Products =3D=3D >> >> Mesos needs to work well with Hadoop, HBase, and other cloud software >> projects. Being hosted on the same infrastructure will facilitate this >> and ultimately help out both Mesos and the projects that can now be >> managed using it. There is, however, a risk that new projects will be bu= ilt >> to run solely on Mesos, introducing a dependency. >> >> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D >> >> While we respect the reputation of the Apache brand and have no doubts t= hat >> it will attract contributors and users, our interest is primarily to giv= e >> Mesos a solid home as an open source project following an established >> development model. Locating the project in Apache will also facilitate >> collaboration with Hadoop, HBase, and other Apache cluster computing >> projects, as discussed in the Alignment section. >> >> >> >> =3D Documentation =3D >> >> Information about Mesos can be found at http://mesos.berkeley.edu. >> The following sources may be useful to start with: >> >> =C2=A0* Documentation for GitHub release: http://github.com/mesos/mesos/= wiki >> =C2=A0* Presentation at Hadoop User Group: >> http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf >> =C2=A0* Tech report on system design and current features: >> http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI >> 2011 conference) >> >> >> >> =3D Initial Source =3D >> >> Mesos has been under development since spring 2009 by a team of graduate >> students and researchers. It is currently hosted on GitHub under a BSD >> license at http://github.com/mesos/mesos. >> >> >> >> =3D External Dependencies =3D >> >> The dependencies all have Apache compatible licenses, including BSD, MIT= , >> Boost, and Apache 2.0. >> >> >> >> =3D Cryptography =3D >> >> Not applicable. >> >> >> >> =3D Required Resources =3D >> >> =3D=3D Mailing Lists =3D=3D >> >> =C2=A0* mesos-private for private PMC discussions (with moderated subscr= iptions) >> =C2=A0* mesos-dev >> =C2=A0* mesos-commits >> =C2=A0* mesos-user >> >> >> >> =3D=3D Subversion Directory =3D=3D >> >> https://svn.apache.org/repos/asf/incubator/mesos >> >> >> >> =3D=3D Issue Tracking =3D=3D >> >> JIRA Mesos (MESOS) >> >> >> >> =3D=3D Other Resources =3D=3D >> >> The existing code already has unit tests, so we would like a Hudson inst= ance >> to run them whenever a new patch is submitted. This can be added after >> project >> creation. >> >> >> >> =3D Initial Committers =3D >> >> =C2=A0* Ali Ghodsi (ali at sics dot se) >> =C2=A0* Benjamin Hindman (benh at eecs dot berkeley dot edu) >> =C2=A0* Andy Konwinski (andyk at eecs dot berkeley dot edu) >> =C2=A0* Matei Zaharia (matei at apache dot org) >> >> A CLA is already on file for Matei Zaharia. >> >> >> =3D Affiliations =3D >> >> =C2=A0* Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science) >> =C2=A0* Benjamin Hindman (UC Berkeley) >> =C2=A0* Andy Konwinski (UC Berkeley) >> =C2=A0* Matei Zaharia (UC Berkeley) >> >> >> >> =3D Sponsors =3D >> >> =3D=3D Champion =3D=3D >> >> Tom White >> >> =3D=3D Nominated Mentors =3D=3D >> >> =C2=A0* Dhruba Borthakur >> =C2=A0* Brian McCallister >> =C2=A0* Tom White >> >> =3D=3D Sponsoring Entity =3D=3D >> >> Incubator PMC >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org >> For additional commands, e-mail: general-help@incubator.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org > For additional commands, e-mail: general-help@incubator.apache.org > > --=20 Best Regards, Edward J. Yoon edwardyoon@apache.org http://blog.udanax.org --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.org