incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Trivial Update of "MyriadProposal" by TedDunning
Date Wed, 18 Feb 2015 15:42:37 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "MyriadProposal" page has been changed by TedDunning:
https://wiki.apache.org/incubator/MyriadProposal?action=diff&rev1=2&rev2=3

Comment:
Reformatted to match wiki-style

- Apache Myriad Proposal
+ = Myriad Proposal =
  
- * Abstract
+ == Abstract ==
  Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together
  on the same cluster and allows dynamic resource allocations across both
  Hadoop and other applications running on the same physical data center
  infrastructure.
  
- * Proposal
+ == Proposal ==
  The vision of Myriad is to provide a comprehensive framework to ensure
  Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
  on either side and prevent the static fragmentation of data center
  resources.
  
- * Background
+ == Background ==
  Project Myriad is the first resource management framework that allows big
  data developers to run YARN-based Hadoop jobs alongside other applications
  and services in production. ebay Inc., MapR, and Mesosphere jointly built
@@ -24, +24 @@

  operational efficiency. Several companies including Twitter have expressed
  interest in Myriad and have begun testing it.
  
- * Rationale
+ == Rationale ==
  Many Hadoop users are building larger clusters (data lake/data hub
  architectures) that support multiple workloads - made possible by the
  advent of Apache Hadoop YARN. As the clusters grow in size and importance,
@@ -39, +39 @@

  on the same physical data center resources, reducing fragmentation of data
  center resources.
  
- * Project Goals
+ == Project Goals ==
  
- ** Initial Goals
+ === Initial Goals ===
- 
- - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
+  * Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
  based allocation of data center resources across Apache Hadoop and other
  distributed applications
- 
- - Ensure YARN based execution frameworks work without any changes when
+  * Ensure YARN based execution frameworks work without any changes when
  running alongside Myriad. YARN Applications will continue to interact and
  run on top of YARN and can choose to be unaware of Myriad.
- 
- - Ensure Mesos based execution frameworks work without any changes when
+  * Ensure Mesos based execution frameworks work without any changes when
  running alongside Myriad. Mesos applications will continue to interact and
  run on Mesos and can choose to be unaware of Myriad.
- 
- - Provide isolation for multi-tenancy.
+  * Provide isolation for multi-tenancy.
- 
-   - Use linux cgroups (and optionally Docker-like technologies to ease
+    * Use linux cgroups (and optionally Docker-like technologies to ease
  packaging, deployment and broader isolation) so that multiple YARN clusters
  can run in their own space and are isolated from each other. YARN’s RM and
  NMs are dockerized.
- 
- - Myriad should be able to manage full YARN lifecycle:
+  * Myriad should be able to manage full YARN lifecycle:
- 
-   - Bring up YARN (RM, NM)
+    * Bring up YARN (RM, NM)
- 
-   - Scale Up/Down YARN
+    * Scale Up/Down YARN
- 
-   - Release resources and shut down YARN
+    * Release resources and shut down YARN
  
- 
- ** Longer Term Goals
+ === Longer Term Goals ===
- 
- - Allow fine-grained dynamic allocation of resources to Hadoop including
+  * Allow fine-grained dynamic allocation of resources to Hadoop including
  the ability to scale up and scale down the cluster.
- 
-   - Provide different policies to allow downsizing running applications on
+    * Provide different policies to allow downsizing running applications on
  Hadoop when resources are taken away from it.
- 
-   - Provide a framework so the downsizing policy is pluggable and users can
+    * Provide a framework so the downsizing policy is pluggable and users can
  write their own implementations.
- 
- - Allow multiple versions of Apache Hadoop to run on the same physical
+  * Allow multiple versions of Apache Hadoop to run on the same physical
  infrastructure
- 
- - Allow workload portability - ability to migrate YARN workloads across
+  * Allow workload portability - ability to migrate YARN workloads across
  various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
- 
- - Security:
+  * Security:
- 
-   - Authentication Requirements:
+    * Authentication Requirements:
- 
-     - Support basic CRAM-MD5 password authentication between Myriad and
+      * Support basic CRAM-MD5 password authentication between Myriad and
  Mesos. Additional authentication mechanisms may be supported in the future.
- 
-     - Traditional user authentication with Hadoop’s HTTP web-consoles
+      * Traditional user authentication with Hadoop’s HTTP web-consoles
  should work as usual.
- 
-   - Authorization:
+    * Authorization:
- 
-     - Only authorized users are allowed to launch YARN clusters.  Mesos
+      * Only authorized users are allowed to launch YARN clusters.  Mesos
  allows to specify which framework principal is allowed to register as a
  particular role.
- 
-   - Encryption on wire:
+    * Encryption on wire:
- 
-     - All control traffic to/from Myriad/Mesos
+      * All control traffic to/from Myriad/Mesos
- 
- - Logs
+  * Logs
- 
-   - Audits (where to store them)
+    * Audits (where to store them)
- 
-     - Log all major activities/events with audit trail - who, what, when,
+      * Log all major activities/events with audit trail - who, what, when,
  result
- 
-     - Launching YARN/RM
+      * Launching YARN/RM
- 
-     - Launching NM’s
+      * Launching NM’s
- 
-     - Downsizing NM’s
+      * Downsizing NM’s
- 
-     - Terminating YARN/RM
+      * Terminating YARN/RM
- 
-   - What to do with old logs?
+    * What to do with old logs?
- 
-   - Debuggability/Visibility
+    * Debuggability/Visibility
- 
-     - Hooks to identify different YARN cluster lifecycles (yarn-id?)
+      * Hooks to identify different YARN cluster lifecycles (yarn-id?)
- 
- - GUI: Capability to scale-up and scale-down by selecting nodes and
+  * GUI: Capability to scale-up and scale-down by selecting nodes and
  providing a scale-up/scale-down factor.
  
- * Architectural Overview
+ == Architectural Overview ==
  
  The following diagram illustrates the high level architecture. YARN (with
  Myriad) is registered as a framework with Mesos master along with possibly
@@ -143, +109 @@

  See
  https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
  
- * Current Status
+ == Current Status ==
- 
  Myriad is under active development. Key components of Myriad are:
  
- ** Myriad Resource Manager (RM) Plugin
+ === Myriad Resource Manager (RM) Plugin ===
- 
- - Plugs into Resource Manager Java process via yarn-site.xml configuration.
+  * Plugs into Resource Manager Java process via yarn-site.xml configuration.
- 
- - Registers Myriad as a framework with Mesos. Receives resource offers from
+  * Registers Myriad as a framework with Mesos. Receives resource offers from
  Mesos.
- 
- - Monitors YARN’s application pipeline and scheduling events to drive
+  * Monitors YARN’s application pipeline and scheduling events to drive
  scale-up or scale-down decisions for Hadoop.
- 
- - Exposes REST APIs to help admins control Hadoop/YARN’s resource
+  * Exposes REST APIs to help admins control Hadoop/YARN’s resource
  consumption. Currently the following APIs are supported:
- 
-   - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
+    * Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
- 
-   - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
+    * Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
  capacity”)
  
- ** Myriad Mesos Executor
+ === Myriad Mesos Executor ===
- 
- - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
+  * Launched on a Mesos slave node by Myriad RM plugin via Mesos.
- 
- - Responsible for launching Node Manager process with appropriate
+  * Responsible for launching Node Manager process with appropriate
  capacities configured in yarn-site.xml.
- 
- - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
+  * Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
  YARN’s cgroups are enabled.
  
  Currently, a working prototype/demo had been built for the goals listed
@@ -180, +136 @@

  at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
  production use.
  
+ === Meritocracy ===
- ** Meritocracy
- 
  We plan to invest in supporting a meritocracy. We will discuss the
  requirements in a public forum. Several companies have already expressed
  interest in this project, and we intend to invite developers to contribute
  and gain karma. We will encourage and monitor community participation so
  that privileges can be extended to those that contribute.
  
+ === Community ===
- ** Community
- 
  We are happy to report that there are existing Apache committers and
  corporate users who are closely involved in the project already. We hope to
  extend the user and developer base further in the future and build a solid
  open source community around Myriad, growing the community and adding
  committers following the Apache Way.
  
- ** Core Developers
+ === Core Developers ===
- 
  The initial technology was built independently by ebay and MapR. ebay built
  the technology in consultation with Ben Hindman. MapR built a working
  prototype in tight consultation and mentorship with Mesosphere.
  
+ === Alignment ===
- ** Alignment
- 
  The initial committers strongly believe that Apache Hadoop YARN and Apache
  Mesos will gain broad adoption and therefore a framework to allow for a
  co-existence of these frameworks that is transparent to applications
  written for YARN and Mesos will serve the needs of the broader community.
  
- * Known Risks
+ == Known Risks ==
  
- ** Inexperience with Open Source
+ === Inexperience with Open Source ===
- 
  Initial Myriad committers have varying levels of experience using and
  contributing to Open Source projects, however by working with our mentors
  and the Apache community we believe we will be able to conduct ourselves in
@@ -221, +172 @@

  awareness of the incubation process and a willingness to embrace The Apache
  Way.
  
- ** Homogenous Developers
+ === Homogenous Developers ===
- 
  There is already diversity in the core developer community as they are
  employed by three different and independent companies viz. ebay inc., MapR,
  and Mesosphere. However, there will continue to be an emphasis on
  increasing the diversity of the developer community.
  
- ** Reliance on Salaried Developers
+ === Reliance on Salaried Developers ===
- 
  Currently, the core developers are paid to work on Myriad. However, once
  the project has a community built around it, we expect to get committers,
  contributors and community from outside the current participating
  organizations.
  
- ** Relationships with Other Apache Products
+ === Relationships with Other Apache Products ===
- 
  Myriad implements interfaces from both Apache YARN and Apache Mesos, and
  requires both to be present so that Myriad can coordinate dynamic resource
  sharing between the two.
  
- ** An Excessive Fascination with the Apache Brand
+ === An Excessive Fascination with the Apache Brand ===
- 
  While we respect the reputation of the Apache brand and have no doubts that
  it will attract contributors and users, our interest is primarily to give
  Myriad a solid home as an open source project following an established
  development model. We have also given reasons in the Rationale and
  Alignment sections.
  
- * Documentation
+ == Documentation ==
- 
  Documentation is included in a docs directory of the repository (See
  https://github.com/mesos/myriad/tree/phase1/docs), and currently details
  how Myriad works, developing the project, auto-scaling a YARN cluster, the
  Myriad REST API, and more. We will improve docs at every revision drop.
  
- * Initial Source
+ == Initial Source ==
- 
  The Myriad codebase has been posted on GitHub for review and licensed under
  an Apache v2 license.
  
  https://github.com/mesos/myriad
  
- * Source and IP Submission Plan
+ == Source and IP Submission Plan ==
- 
  During incubation, the codebase will be available at
  https://github.com/apache/incubator-myriad/ and contributors will commit
  appropriate contribute license agreements.
  
- * External Dependencies
+ == External Dependencies ==
- 
  All Myriad dependencies have Apache compatible licenses.
  
- * Cryptography
+ == Cryptography ==
- 
  Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however,
  use standard API’s and tools for SSH And SSL communication where necessary.
  
- * Required Resources
+ == Required Resources ==
  
- ** Mailing Lists
+ === Mailing Lists ===
- 
- - myriad-private for private PMC conversations
+  * myriad-private for private PMC conversations
- 
- - myriad-dev
+  * myriad-dev
- 
- - myriad-commits
+  * myriad-commits
  
- - myriad-user
- 
- ** Version Control
+ === Version Control ===
- 
- We prefer to use Git as our source control system: git://
+ We prefer to use Git as our source control system:
- git.apache.org/myriad
+ git://git.apache.org/myriad
  
- ** Issue Tracking
+ === Issue Tracking ===
- 
  JIRA Myriad (MYRIAD)
  
- * Initial Committers
+ == Initial Committers ==
- 
- - Santosh Marella (smarella at mapr dot com)
+  * Santosh Marella (smarella at mapr dot com)
- 
- - Mohit Soni (mohitsoni1989 at gmail dot com)
+  * Mohit Soni (mohitsoni1989 at gmail dot com)
- 
- - Adam Bordelon (me at apache dot org) *
+  * Adam Bordelon (me at apache dot org) *
- 
- - Meghdoot Bhattacharya  ( mbhattacharya at paypal dot com)
+  * Meghdoot Bhattacharya  ( mbhattacharya at paypal dot com)
- 
- - Anoop Dawar (anoopdawar at gmail dot com)
+  * Anoop Dawar (anoopdawar at gmail dot com)
- 
- - Jim Scott (jim at 13ways dot com)
+  * Jim Scott (jim at 13ways dot com)
- 
- - Ken Sipe (kensipe at gmail dot com)
+  * Ken Sipe (kensipe at gmail dot com)
  
- * Affiliations
+ == Affiliations ==
- 
- - Santosh Marella, MapR
+  * Santosh Marella, MapR
- 
- - Mohit Soni, ebay Inc.
+  * Mohit Soni, ebay Inc.
- 
- - Adam Bordelon, Mesosphere
+  * Adam Bordelon, Mesosphere
- 
- - Meghdoot Bhattacharya, ebay Inc.
+  * Meghdoot Bhattacharya, ebay Inc.
- 
- - Anoop Dawar, MapR
+  * Anoop Dawar, MapR
- 
- - Jim Scott, MapR
+  * Jim Scott, MapR
- 
- - Ken Sipe, Mesosphere
+  * Ken Sipe, Mesosphere
  
- * Sponsors
+ == Sponsors ==
  
- ** Champion (Proposal)
+ === Champion (Proposal) ===
- 
- - Ben Hindman (benh at apache dot org)
+  * Ben Hindman (benh at apache dot org)
  
- ** Nominated Mentors
+ === Nominated Mentors ===
- 
- - Ben Hindman (benh at apache dot org) - Mesosphere
+  * Ben Hindman (benh at apache dot org) - Mesosphere
- 
- - Danese Cooper (danese at apache dot org) - ebay, Inc.
+  * Danese Cooper (danese at apache dot org) - ebay, Inc.
- 
- - Ted Dunning (tdunning at apache dot org) - MapR
+  * Ted Dunning (tdunning at apache dot org) - MapR
  
- ** Sponsoring Entity
+ === Sponsoring Entity ===
- 
- Apache Incubator
+  * Apache Incubator
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message