Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <4D0D2857.8000100@apache.org>
References: <4D0D2857.8000100@apache.org>
From: Tom White <tomwhite@apache.org>
Date: Mon, 20 Dec 2010 14:53:07 -0800
Message-ID: <AANLkTimWYKVa6PN00LsOqh0F5p0-yN9EdJJbaWGKrYv=@mail.gmail.com>
Subject: Re: [VOTE] Mesos to enter the incubator
To: general@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

Tom

On Sat, Dec 18, 2010 at 1:32 PM, Matei Zaharia <matei@apache.org> wrote:
> We've finalized our proposal for Mesos
> (http://wiki.apache.org/incubator/MesosProposal) and we'd now like to put=
 it
> up for vote.
>
> I'll tally the results after five days, on December 23rd.
>
> Thanks,
>
> Matei
>
>
>
>
> ---------------------------------------------------------------------
> MESOS PROPOSAL
> ---------------------------------------------------------------------
>
>
> =3D Abstract =3D
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> =3D Proposal =3D
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> =3D Background =3D
>
> Mesos was inspired by operational issues experienced in large Apache Hado=
op
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce
> clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a
> layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has
> attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> =3D Rationale =3D
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally=
.
> To address these challenges, Mesos is designed around two principles:
>
> =A0* Fine-grained sharing: Mesos allocates resources at the level of "tas=
ks"
> =A0 within a job, allowing applications to scale up and down over time an=
d
> =A0 to take turns accessing data on cluster nodes.
> =A0* Application-controlled scheduling: Applications control which nodes
> =A0 their tasks run on, allowing them to achieve placement goals such as
> =A0 data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs a=
n
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> =3D Current Status =3D
>
> =3D=3D Meritocracy =3D=3D
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> =3D=3D Community =3D=3D
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the
> user
> and developer base further in the future. The current developers and user=
s
> are all interested in building a solid open source community around Mesos=
.
>
> To work towards an open source community, we have been using the GitHub
> issue
> tracker and mailing lists at Berkeley for development discussions within =
our
> group for several months now.
>
> =3D=3D Core Developers =3D=3D
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin
> Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started =
as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apac=
he
> Hadoop
> (several scheduling algorithms).
>
> =3D=3D Alignment =3D=3D
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide
> range
> of applications beyond Hadoop as well. Mesos complements the existing Apa=
che
> cloud computing projects by providing a unified way to manage these syste=
ms
> and to share resources and data between them.
>
>
>
> =3D Known Risks =3D
>
> =3D=3D Orphaned Products =3D=3D
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects=
.
> However, because of the broad scope of Mesos, we all plan to continue
> working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> =3D=3D Inexperience with Open Source =3D=3D
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopte=
d
> by a Swedish company.
>
> =3D=3D Homogeneous Developers =3D=3D
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> =3D=3D Reliance on Salaried Developers =3D=3D
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rathe=
r
> than for the sake of employment. We all intend to continue working on Mes=
os
> as volunteers.
>
> =3D=3D Relationships with Other Apache Products =3D=3D
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be bui=
lt
> to run solely on Mesos, introducing a dependency.
>
> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D
>
> While we respect the reputation of the Apache brand and have no doubts th=
at
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing
> projects, as discussed in the Alignment section.
>
>
>
> =3D Documentation =3D
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> =A0* Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> =A0* Presentation at Hadoop User Group:
> http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> =A0* Tech report on system design and current features:
> http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI
> 2011 conference)
>
>
>
> =3D Initial Source =3D
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> =3D External Dependencies =3D
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> =3D Cryptography =3D
>
> Not applicable.
>
>
>
> =3D Required Resources =3D
>
> =3D=3D Mailing Lists =3D=3D
>
> =A0* mesos-private for private PMC discussions (with moderated subscripti=
ons)
> =A0* mesos-dev
> =A0* mesos-commits
> =A0* mesos-user
>
>
>
> =3D=3D Subversion Directory =3D=3D
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> =3D=3D Issue Tracking =3D=3D
>
> JIRA Mesos (MESOS)
>
>
>
> =3D=3D Other Resources =3D=3D
>
> The existing code already has unit tests, so we would like a Hudson insta=
nce
> to run them whenever a new patch is submitted. This can be added after
> project
> creation.
>
>
>
> =3D Initial Committers =3D
>
> =A0* Ali Ghodsi (ali at sics dot se)
> =A0* Benjamin Hindman (benh at eecs dot berkeley dot edu)
> =A0* Andy Konwinski (andyk at eecs dot berkeley dot edu)
> =A0* Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> =3D Affiliations =3D
>
> =A0* Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> =A0* Benjamin Hindman (UC Berkeley)
> =A0* Andy Konwinski (UC Berkeley)
> =A0* Matei Zaharia (UC Berkeley)
>
>
>
> =3D Sponsors =3D
>
> =3D=3D Champion =3D=3D
>
> Tom White
>
> =3D=3D Nominated Mentors =3D=3D
>
> =A0* Dhruba Borthakur
> =A0* Brian McCallister
> =A0* Tom White
>
> =3D=3D Sponsoring Entity =3D=3D
>
> Incubator PMC
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org