incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Trivial Update of "MyriadProposal" by Adam
Date Wed, 18 Feb 2015 20:07:12 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "MyriadProposal" page has been changed by Adam:
https://wiki.apache.org/incubator/MyriadProposal?action=diff&rev1=3&rev2=4

Comment:
Fix wrapping and indentation

  = Myriad Proposal =
- 
  == Abstract ==
+ Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together on the same
cluster and allows dynamic resource allocations across both Hadoop and other applications
running on the same physical data center infrastructure.
- Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together
- on the same cluster and allows dynamic resource allocations across both
- Hadoop and other applications running on the same physical data center
- infrastructure.
  
  == Proposal ==
+ The vision of Myriad is to provide a comprehensive framework to ensure Apache Hadoop YARN
and Apache Mesos can interoperate with minimal changes on either side and prevent the static
fragmentation of data center resources.
- The vision of Myriad is to provide a comprehensive framework to ensure
- Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
- on either side and prevent the static fragmentation of data center
- resources.
  
  == Background ==
+ Project Myriad is the first resource management framework that allows big data developers
to run YARN-based Hadoop jobs alongside other applications and services in production. ebay
Inc., MapR, and Mesosphere jointly built Myriad (available on Github at https://github.com/mesos/myriad)
with the vision of freeing big data jobs from siloed clusters and consolidating infrastructure
into a single pool of resources for greater utilization and operational efficiency. Several
companies including Twitter have expressed interest in Myriad and have begun testing it.
- Project Myriad is the first resource management framework that allows big
- data developers to run YARN-based Hadoop jobs alongside other applications
- and services in production. ebay Inc., MapR, and Mesosphere jointly built
- Myriad (available on Github at https://github.com/mesos/myriad) with the
- vision of freeing big data jobs from siloed clusters and consolidating
- infrastructure into a single pool of resources for greater utilization and
- operational efficiency. Several companies including Twitter have expressed
- interest in Myriad and have begun testing it.
  
  == Rationale ==
+ Many Hadoop users are building larger clusters (data lake/data hub architectures) that support
multiple workloads - made possible by the advent of Apache Hadoop YARN. As the clusters grow
in size and importance, they become an important application within the broader datacenter.
At the same time, Apache Mesos enables efficient resource isolation and sharing across distributed
applications for the broader data center, for instance MPI, Spark, long running web services,
build/test infrastructure, traditional linux applications/scripts, and others (including arbitrary
docker images).
- Many Hadoop users are building larger clusters (data lake/data hub
- architectures) that support multiple workloads - made possible by the
- advent of Apache Hadoop YARN. As the clusters grow in size and importance,
- they become an important application within the broader datacenter. At the
- same time, Apache Mesos enables efficient resource isolation and sharing
- across distributed applications for the broader data center, for instance
- MPI, Spark, long running web services, build/test infrastructure,
- traditional linux applications/scripts, and others (including arbitrary
- docker images).
  
+ Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos on the same physical
data center resources, reducing fragmentation of data center resources.
- Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
- on the same physical data center resources, reducing fragmentation of data
- center resources.
  
  == Project Goals ==
+ === Initial Goals ===
+  * Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy based allocation
of data center resources across Apache Hadoop and other distributed applications
  
+  * Ensure YARN based execution frameworks work without any changes when running alongside
Myriad. YARN Applications will continue to interact and run on top of YARN and can choose
to be unaware of Myriad.
+ 
+  * Ensure Mesos based execution frameworks work without any changes when running alongside
Myriad. Mesos applications will continue to interact and run on Mesos and can choose to be
unaware of Myriad.
+ 
- === Initial Goals ===
-  * Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
- based allocation of data center resources across Apache Hadoop and other
- distributed applications
-  * Ensure YARN based execution frameworks work without any changes when
- running alongside Myriad. YARN Applications will continue to interact and
- run on top of YARN and can choose to be unaware of Myriad.
-  * Ensure Mesos based execution frameworks work without any changes when
- running alongside Myriad. Mesos applications will continue to interact and
- run on Mesos and can choose to be unaware of Myriad.
   * Provide isolation for multi-tenancy.
+   * Use linux cgroups (and optionally Docker-like technologies to ease packaging, deployment
and broader isolation) so that multiple YARN clusters can run in their own space and are isolated
from each other. YARN’s RM and NMs are dockerized.
+ 
-    * Use linux cgroups (and optionally Docker-like technologies to ease
- packaging, deployment and broader isolation) so that multiple YARN clusters
- can run in their own space and are isolated from each other. YARN’s RM and
- NMs are dockerized.
   * Myriad should be able to manage full YARN lifecycle:
-    * Bring up YARN (RM, NM)
+   * Bring up YARN (RM, NM)
-    * Scale Up/Down YARN
+   * Scale Up/Down YARN
-    * Release resources and shut down YARN
+   * Release resources and shut down YARN
  
  === Longer Term Goals ===
-  * Allow fine-grained dynamic allocation of resources to Hadoop including
- the ability to scale up and scale down the cluster.
+  * Allow fine-grained dynamic allocation of resources to Hadoop including the ability to
scale up and scale down the cluster.
+ 
-    * Provide different policies to allow downsizing running applications on
+  * Provide different policies to allow downsizing running applications on Hadoop when resources
are taken away from it.
- Hadoop when resources are taken away from it.
+ 
-    * Provide a framework so the downsizing policy is pluggable and users can
+  * Provide a framework so the downsizing policy is pluggable and users can write their own
implementations.
- write their own implementations.
+ 
-  * Allow multiple versions of Apache Hadoop to run on the same physical
+  * Allow multiple versions of Apache Hadoop to run on the same physical infrastructure
+ 
+  * Allow workload portability - ability to migrate YARN workloads across various cloud infrastructures
seamlessly (e.g. GCE, AWS, etc)
+ 
- infrastructure
-  * Allow workload portability - ability to migrate YARN workloads across
- various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
-  * Security:
-    * Authentication Requirements:
+  * Authentication Requirements:
+   * Support basic CRAM-MD5 password authentication between Myriad and Mesos. Additional
authentication mechanisms may be supported in the future.
-      * Support basic CRAM-MD5 password authentication between Myriad and
- Mesos. Additional authentication mechanisms may be supported in the future.
-      * Traditional user authentication with Hadoop’s HTTP web-consoles
+   * Traditional user authentication with Hadoop’s HTTP web-consoles should work as usual.
- should work as usual.
+ 
-    * Authorization:
+  * Authorization:
+   * Only authorized users are allowed to launch YARN clusters.  Mesos allows to specify
which framework principal is allowed to register as a particular role.
+ 
-      * Only authorized users are allowed to launch YARN clusters.  Mesos
- allows to specify which framework principal is allowed to register as a
- particular role.
-    * Encryption on wire:
+  * Encryption on wire:
-      * All control traffic to/from Myriad/Mesos
+   * All control traffic to/from Myriad/Mesos
+ 
   * Logs
-    * Audits (where to store them)
+   * Audits (where to store them)
-      * Log all major activities/events with audit trail - who, what, when,
+    * Log all major activities/events with audit trail - who, what, when, result
- result
-      * Launching YARN/RM
+   * Launching YARN/RM
-      * Launching NM’s
+   * Launching NM’s
-      * Downsizing NM’s
+   * Downsizing NM’s
-      * Terminating YARN/RM
+   * Terminating YARN/RM
-    * What to do with old logs?
+   * What to do with old logs?
+ 
-    * Debuggability/Visibility
+  * Debuggability/Visibility
-      * Hooks to identify different YARN cluster lifecycles (yarn-id?)
+   * Hooks to identify different YARN cluster lifecycles (yarn-id?)
+ 
-  * GUI: Capability to scale-up and scale-down by selecting nodes and
+  * GUI: Capability to scale-up and scale-down by selecting nodes and providing a scale-up/scale-down
factor.
- providing a scale-up/scale-down factor.
  
  == Architectural Overview ==
+ The following diagram illustrates the high level architecture. YARN (with Myriad) is registered
as a framework with Mesos master along with possibly other Mesos frameworks. This enables
YARN to share cluster resources with other Mesos frameworks providing elasticity of resources
between Hadoop workloads and Mesos frameworks.
  
- The following diagram illustrates the high level architecture. YARN (with
- Myriad) is registered as a framework with Mesos master along with possibly
- other Mesos frameworks. This enables YARN to share cluster resources with
- other Mesos frameworks providing elasticity of resources between Hadoop
- workloads and Mesos frameworks.
- 
- See
- https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
+ See https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
  
  == Current Status ==
  Myriad is under active development. Key components of Myriad are:
  
  === Myriad Resource Manager (RM) Plugin ===
   * Plugs into Resource Manager Java process via yarn-site.xml configuration.
-  * Registers Myriad as a framework with Mesos. Receives resource offers from
+  * Registers Myriad as a framework with Mesos. Receives resource offers from Mesos.
- Mesos.
+ 
-  * Monitors YARN’s application pipeline and scheduling events to drive
+  * Monitors YARN’s application pipeline and scheduling events to drive scale-up or scale-down
decisions for Hadoop.
- scale-up or scale-down decisions for Hadoop.
-  * Exposes REST APIs to help admins control Hadoop/YARN’s resource
- consumption. Currently the following APIs are supported:
+ 
+  * Exposes REST APIs to help admins control Hadoop/YARN’s resource consumption. Currently
the following APIs are supported:
+ 
-    * Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
+  * Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
-    * Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
+  * Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU capacity”)
- capacity”)
  
  === Myriad Mesos Executor ===
   * Launched on a Mesos slave node by Myriad RM plugin via Mesos.
-  * Responsible for launching Node Manager process with appropriate
+  * Responsible for launching Node Manager process with appropriate capacities configured
in yarn-site.xml.
- capacities configured in yarn-site.xml.
-  * Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
- YARN’s cgroups are enabled.
  
+  * Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case YARN’s cgroups
are enabled.
+ 
+ Currently, a working prototype/demo had been built for the goals listed under the “Initial
Goals” section. Open issues and enhancements are tracked at https://github.com/mesos/myriad/issues.
Myriad is not yet tested for production use.
- Currently, a working prototype/demo had been built for the goals listed
- under the “Initial Goals” section. Open issues and enhancements are tracked
- at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
- production use.
  
  === Meritocracy ===
+ We plan to invest in supporting a meritocracy. We will discuss the requirements in a public
forum. Several companies have already expressed interest in this project, and we intend to
invite developers to contribute and gain karma. We will encourage and monitor community participation
so that privileges can be extended to those that contribute.
- We plan to invest in supporting a meritocracy. We will discuss the
- requirements in a public forum. Several companies have already expressed
- interest in this project, and we intend to invite developers to contribute
- and gain karma. We will encourage and monitor community participation so
- that privileges can be extended to those that contribute.
  
  === Community ===
+ We are happy to report that there are existing Apache committers and corporate users who
are closely involved in the project already. We hope to extend the user and developer base
further in the future and build a solid open source community around Myriad, growing the community
and adding committers following the Apache Way.
- We are happy to report that there are existing Apache committers and
- corporate users who are closely involved in the project already. We hope to
- extend the user and developer base further in the future and build a solid
- open source community around Myriad, growing the community and adding
- committers following the Apache Way.
  
  === Core Developers ===
+ The initial technology was built independently by ebay and MapR. ebay built the technology
in consultation with Ben Hindman. MapR built a working prototype in tight consultation and
mentorship with Mesosphere.
- The initial technology was built independently by ebay and MapR. ebay built
- the technology in consultation with Ben Hindman. MapR built a working
- prototype in tight consultation and mentorship with Mesosphere.
  
  === Alignment ===
+ The initial committers strongly believe that Apache Hadoop YARN and Apache Mesos will gain
broad adoption and therefore a framework to allow for a co-existence of these frameworks that
is transparent to applications written for YARN and Mesos will serve the needs of the broader
community.
- The initial committers strongly believe that Apache Hadoop YARN and Apache
- Mesos will gain broad adoption and therefore a framework to allow for a
- co-existence of these frameworks that is transparent to applications
- written for YARN and Mesos will serve the needs of the broader community.
  
  == Known Risks ==
- 
  === Inexperience with Open Source ===
+ Initial Myriad committers have varying levels of experience using and contributing to Open
Source projects, however by working with our mentors and the Apache community we believe we
will be able to conduct ourselves in accordance with Apache Incubator guidelines. The close
relationship between the Myriad team and Apache Mesos and Apache Hadoop means there is an
awareness of the incubation process and a willingness to embrace The Apache Way.
- Initial Myriad committers have varying levels of experience using and
- contributing to Open Source projects, however by working with our mentors
- and the Apache community we believe we will be able to conduct ourselves in
- accordance with Apache Incubator guidelines. The close relationship between
- the Myriad team and Apache Mesos and Apache Hadoop means there is an
- awareness of the incubation process and a willingness to embrace The Apache
- Way.
  
  === Homogenous Developers ===
+ There is already diversity in the core developer community as they are employed by three
different and independent companies viz. ebay inc., MapR, and Mesosphere. However, there will
continue to be an emphasis on increasing the diversity of the developer community.
- There is already diversity in the core developer community as they are
- employed by three different and independent companies viz. ebay inc., MapR,
- and Mesosphere. However, there will continue to be an emphasis on
- increasing the diversity of the developer community.
  
  === Reliance on Salaried Developers ===
+ Currently, the core developers are paid to work on Myriad. However, once the project has
a community built around it, we expect to get committers, contributors and community from
outside the current participating organizations.
- Currently, the core developers are paid to work on Myriad. However, once
- the project has a community built around it, we expect to get committers,
- contributors and community from outside the current participating
- organizations.
  
  === Relationships with Other Apache Products ===
+ Myriad implements interfaces from both Apache YARN and Apache Mesos, and requires both to
be present so that Myriad can coordinate dynamic resource sharing between the two.
- Myriad implements interfaces from both Apache YARN and Apache Mesos, and
- requires both to be present so that Myriad can coordinate dynamic resource
- sharing between the two.
  
  === An Excessive Fascination with the Apache Brand ===
+ While we respect the reputation of the Apache brand and have no doubts that it will attract
contributors and users, our interest is primarily to give Myriad a solid home as an open source
project following an established development model. We have also given reasons in the Rationale
and Alignment sections.
- While we respect the reputation of the Apache brand and have no doubts that
- it will attract contributors and users, our interest is primarily to give
- Myriad a solid home as an open source project following an established
- development model. We have also given reasons in the Rationale and
- Alignment sections.
  
  == Documentation ==
+ Documentation is included in a docs directory of the repository (See https://github.com/mesos/myriad/tree/phase1/docs),
and currently details how Myriad works, developing the project, auto-scaling a YARN cluster,
the Myriad REST API, and more. We will improve docs at every revision drop.
- Documentation is included in a docs directory of the repository (See
- https://github.com/mesos/myriad/tree/phase1/docs), and currently details
- how Myriad works, developing the project, auto-scaling a YARN cluster, the
- Myriad REST API, and more. We will improve docs at every revision drop.
  
  == Initial Source ==
- The Myriad codebase has been posted on GitHub for review and licensed under
+ The Myriad codebase has been posted on GitHub for review and licensed under an Apache v2
license.
- an Apache v2 license.
  
  https://github.com/mesos/myriad
  
  == Source and IP Submission Plan ==
+ During incubation, the codebase will be available at https://github.com/apache/incubator-myriad/
and contributors will commit appropriate contribute license agreements.
- During incubation, the codebase will be available at
- https://github.com/apache/incubator-myriad/ and contributors will commit
- appropriate contribute license agreements.
  
  == External Dependencies ==
  All Myriad dependencies have Apache compatible licenses.
  
  == Cryptography ==
+ Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however, use standard
API’s and tools for SSH And SSL communication where necessary.
- Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however,
- use standard API’s and tools for SSH And SSL communication where necessary.
  
  == Required Resources ==
- 
  === Mailing Lists ===
   * myriad-private for private PMC conversations
   * myriad-dev
   * myriad-commits
  
  === Version Control ===
- We prefer to use Git as our source control system:
+ We prefer to use Git as our source control system: git://git.apache.org/myriad
- git://git.apache.org/myriad
  
  === Issue Tracking ===
  JIRA Myriad (MYRIAD)
@@ -253, +169 @@

   * Ken Sipe, Mesosphere
  
  == Sponsors ==
- 
  === Champion (Proposal) ===
   * Ben Hindman (benh at apache dot org)
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message