incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "HoyaProposal" by SteveLoughran
Date Thu, 30 Jan 2014 13:06:25 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "HoyaProposal" page has been changed by SteveLoughran:
https://wiki.apache.org/incubator/HoyaProposal?action=diff&rev1=9&rev2=10

Comment:
reworked Hoya proposal

  
  == Abstract ==
  
+ Hoya is an application to deploy and manage existing distributed applications in a YARN
cluster.
- Hoya is an application to provision and create instances of distributed
- applications across an Apache Hadoop cluster running the YARN resource manager.
  
+ The core concept is : *Hoya understands YARN so your application doesn't need to*.
- It's goal is to be "an extensible tool for creating and managing distributed
- applications under YARN"
- 
  == Proposal ==
  
  Hoya allows users to deploy distributed applications across a Hadoop
- cluster, using the YARN resource manager to allocate and distribute parts of an
+ cluster, using the YARN Resource Manager to allocate and distribute parts of an
- application.. Hoya can monitor the health of these deployed
+ application across the cluster.. Hoya monitors the health of these deployed
+ containers, and react to their failure by creating new instances of them. It supports dynamic
updates to cluster size, and "freezing" and "thawing" the application.
- components, and react to their failure or the loss of the servers on which they
- run on by creating new instances of them. Hoya can expand or shrink the size of
- the application in response to user commands, and pause and resume an
- application. A ''frozen'' application has its state saved to disk, but consumes
- no cluster compute resources until ''thawed''
  
+ In this way, Hoya can take a classic distributed application -such as Apache HBase and Apache
Accumulo- and make it a dynamic YARN application.
- In this way, Hoya can take a classic "deployed by the ops team on every server"
- Hadoop application -such as Apache HBase and Apache Accumulo- and make it a
- service in a YARN cluster which a user can instantiate, which can flex in size
- based on load, and be frozen and thawed on demand.
- 
  
  == Background ==
  
+ Hoya was developed at Hortonworks to support the deployment of HBase clusters in YARN. It
was done both to showcase this possibility, and to drive YARN development towards the needs
of long-lived applications. 
- Hoya was written as an extension of the YARN resource framework: a YARN
- application to deliver the flexibility and recoverability features of a YARN
- application to existing Hadoop applications. The original Groovy-language
- prototype targeted HBase, hence the name Hoya: HBase On YARN
  
+ It has evolved to provide an extension model: providers. These allow Hoya to support different
applications - it now supports Apache Accumulo, and can easily support [[https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md|other
suitable applications]]
- The original Groovy based "Hoya 0.1" application could create an HBase cluster
- in a YARN cluster, using YARN to copy
- over the HBase .tar binaries and configuration files supplied by the end user
- and modified by Hoya en route. Hoya provided its own YARN Application Master,
- (AM), which forked the HBase master as child process, and requested and
- released containers for HBase worker nodes based on demand.
  
+ Hoya is now capable of running long-lived, dynamic applications in YARN clusters. It is
already being used in internal Proof of Concept applications and in testing the applications
and YARN itself; other people and organisations are experimenting with it and providing feedback.
- Key features of this version were
-  *  Automatic download and expansion of {{{.tar}}} and {{{.tar.gz}}} binaries from HDFS
-which allowed for side-by-side clusters of different versions of an application.
-  *  The option to bypass the tarball download and run with pre-installed binaries.
-  *  Patching of {{{hadoop-site.xml}}} for filesystem, zookeeper and port bindings as well
as other user-specified configurations when the application is initially specified. The user
identified a template {{{conf/}}} directory which was snapshotted and then patched.
-  *  Flexing of cluster size -grown or shrunk as requested, optionally persisted for future
runs.
- 
- Since then Hoya has evolved based on the experiences of using the
- previous iterations, and a long-term goal of creating and managing distributed applications
- 
-  * Added the notion of a ''Provider'', a set of classes containing the code to support different
applications; factoring out HBase support into one such provider.
-  * Moved from a simple model of "1 one master, many workers", to one of multiple roles;
each role implementing a different part of a larger  distributed application.
-  * Moved the HBase master to be just another role, allowing HBase clusters to be created
with multiple backup master nodes -for better availability.
-  * Added an Accumulo provider, with support for its many different roles: {{{master}}},
{{{tserver}}}, {{{monitor}}}, {{{tracer}}} and {{{gc}}}.
-  * Added [[https://github.com/hortonworks/hoya/blob/master/src/site/markdown/rolehistory.md|role
placement history ]]. This is quite a sophisticated piece of code -we persist the placement
history to HDFS whenever it changes, then use this to build a list of which nodes to request
containers on. It increases the likelihood that the workers come up on nodes that have the
data, so even if you bring up a small hbase cluster in a large YARN cluster, there's not that
much data to be moved around.
-  * Added initial failure tracking, to track servers that appear to be unreliable, and to
recognize when so many role instances are failing that the Hoya cluster should consider itself
failed
- 
- In the process the core application code was migrated to Java, support for secure Hadoop
clusters added, and the overall feature set and experience of using the tool improved.
  
  == Rationale ==
  
+ Hadoop YARN's cluster manager makes it possible to convert static, one-per-node, cluster-wide
services, into dynamic, user-specific applications, where each project can run their own version.
Resource allocation is more flexible, failure and scalability improved, and new applications
can be developed more easily.
- The Hadoop "stack" has long included applications above the core HDFS and
- MapReduce layers, with the column table database, Apache HBase, one of the key
- applications. To date, an HBase region server has been expected to have been
- deployed on every server in the cluster, managed by the operations team and
- their tools as yet another static part of the Hadoop system. While an ideal
- model for a single-application Hadoop cluster, this model is inflexible in its
- use of cluster resources, and, if a cluster was shared across applications,
- made management of the HBase database complex technically and politically.
- Resource allocations, HBase versions, security access , downtime -all issues
- that would need to be resolved across projects.
  
+ Hoya's goal is to take existing Hadoop applications and host them inside a YARN cluster,
giving them all these benefits - without rewriting the applications.
- It was precisely to address these same problems at the MapReduce that the YARN
- scheduler was created for: to move the MapReduce engine from a pre-installed
- operations-managed service shared across a cluster, to a **user-level
- application** : an applications that individuals could be deployed -with their
- own version, resource allocations and extensions. Hadoop 2's YARN scheduler
- also had a goal that went beyond a better MapReduce experience: it is designed
- to open up the cluster's resources to other applications, such as the Apache-incubating
Samza
- Storm applications that are written with explicit support for YARN.
- 
- The applications that YARN can schedule can stretch beyond analytics
- applications, so allowing mixed-use clusters. Which includes the column table
- databases and other parts of the Hadoop ecosystem. If YARN can deploy them,
- they gain the dynamic deployment, failure recovery and cluster size flexing
- opportunities, while eliminating the "single version per cluster" restriction
- today.
- 
- A rewrite purely for YARN is possible, and it may be something that existing
- projects could consider in future, once Hadoop 1.x support is no longer a need.
- Until then, they need a way to be scheduled and deployed in a YARN cluster
- without being rewritten: Hoya can provide this.
  
  == Initial Goals ==
  
-  1. Donate the (already ASF-licensed) Hoya source code and documentation to the Apache Software
Foundation.
+  1. Donate the Hoya source code and documentation to the Apache Software Foundation.
-  1. Setup and standardize the open governance of the Hoya project.
+  2. Setup and standardize the open governance of the Hoya project.
-  1. Build a user and developer community
+  3. Build a user and developer community
-  1. Tie in better with HBase, Accumulo and other projects that can take advantage of a YARN
cluster -yet which are not ready to be explicitly migrated to YARN.
-  1. Build a system that can be used to migrate more distributed applications into YARN clusters.
+  4. Tie in better with HBase, Accumulo and other projects -both ASF and external- that can
be deployed in a YARN cluster without any code changes.
+  5. Help migrate more distributed applications into YARN clusters - such as Apache HAMA.
  
  == Longer Term Options ==
  
  There are some longer term possibilities that could improve Hoya
  
+  1. Implement a management API for managing Hoya applications by tools such as Apache Ambari.

-  1. A web UI tracking and redirecting to the (changing) location of deployed services.
+  2. Add a web UI for tracking and redirecting to the (changing) location of deployed services.
-  1. Provide a Java API to ease creation and manipulation of Hoya-deployed clusters by other
programs.
+  3. Provide a Java API to ease creation and manipulation of Hoya-deployed clusters by other
programs.
-  1. Adding metrics for management tools.
-  1. Explore/Research sophisticated placement and failure tracking algorithms. Its precisely
because this is a less mature product that we can experiment here.
+  4. Explore/Research sophisticated placement and failure tracking algorithms. Its precisely
because this is a less mature product that we can experiment here.
-  1. Explore load-driven cluster sizing.
+  5. Explore load-driven cluster sizing.
+  6. Explore leveraging Twill internally in Hoya.
  
+ Hoya is driving YARN service support via [[https://issues.apache.org/jira/browse/YARN-896|YARN-896]].
We intend to evolve features and get practical experience using them before merging them into
the Hadoop codebase.
- Hoya is already driving some Hadoop development, such as [[https://issues.apache.org/jira/browse/YARN-896|YARN-896]].
It's also a place where we can evolve features
- and get practical experience using them before submitting them to the Hadoop codebase.
- 
- We may also benefit from changes made to HBase and Accumulo -such as assignment retention
over restart [[https://issues.apache.org/jira/browse/HBASE-10333|HBASE-10333]]. In exchange,
these projects will get extra test suites to explore failure handling and load handling during
dynamic cluster operations -as well as the ability to host multiple instances in a single
- YARN cluster.
- 
  
  == Current Status ==
  
- Hoya is a very young project, six months old, and not yet in production use.
- 
- There is interest in it -especially from people who want to explore Apache
- Accumulo. This shows a key benefit of Hoya: anyone with access to a Hadoop
- cluster can use Hoya to deploy the applications that it manages -here Accumulo.
- Even if the users go on to have a static Accumulo deployment in production, if
- Hoya has helped people get started with the product, it can be considered a
- success.
- 
- == Meritocracy ==
+ === Meritocracy ===
  
  The core of the code was originally written by one person: Steve Loughran, who
  has long-standing experience in Apache projects, with colleagues with HBase
  experience (Ted Yu, Devaraj Das) and Accumulo (Billie Rinaldi, Josh Elser).
  
- 
- == Community ==
+ === Community ===
  
+ We are happy to report that there are folks in Accumulo, HBase and some users outside Hortonworks
who are closely involved in the project already.
- Being a young project there is little community, something github hosting
- does little to develop.
  
+ We hope to extend the user and developer base further in the future and build a solid open
source community around Hoya, growing the community and add committers following the Apache
meritocracy model
- Where there is community is in HBase and Accumulo proper: a challenge for Hoya
- will be how to build awareness of and use of Hoya in these communities. For
- that we have to deliver tangible benefits.
  
+ 
- == Alignment ==
+ === Alignment ===
  
  The project is completely aligned with Apache, from its build process up. It depends on
- Apache Hadoop, it currently deploys HBase and Accumulo.
+ Apache Hadoop, and it currently deploys HBase and Accumulo.
  
- For Hadoop, it, along with Samza, drives the work of supporting long-lived
+ Hoya and Samza are driving the work of supporting long-lived
  services in YARN. While many of these relate to service longevity, there is also the
  challenge of having low-latency table lookups co-exist with CPU-and-IO
+ intensive analytics workloads. 
- intensive analytics workloads. This may drive future developments in Hadoop
- HDFS itself.
  
- 
- === Relationship with Twill ===
+ ==== Relationship with Apache Twill ====
  
+ Twill is a library, a convenience library, that one can use to write YARN applications.
Hoya aims to provide a general framework using which one can take existing applications (HBase/
& Accumulo to start with), and make them run well in a Yarn cluster, without intruding
at all into their internals. 
- The goals of the two projects are different - Twill's
- is to make YARN app developers' lives easier, while Hoya is a
- tool to deploy existing distributed-frameworks easily in a Yarn
- cluster, and be able to later do basic management
- ''without making fundamental changes to the applications themselves''.
  
+ The key differentiators are 
+  * '''Long lived static applications:''' the application's containers are expected to be
relatively stable, with their termination being an unexpected event to which Hoya must react.
+  * '''no application code-changes''':  The only glue between the App and Hoya is a Hoya
interface that the App needs to implement for it to be deployable/manageable by Hoya.
- Things like dynamic configuration patching for
- the applications like HBase to easily run in a Yarn cluster, security
- issues, failure handling and defining a model for reacting to
- failures, being able to store some state about applications to
- facilitate better application restart behavior in a Yarn cluster, etc.
- would be in the purview of Hoya. Management frameworks could use Hoya
- as a tool as well to talk to Yarn and do application
- start/stop/shrink/expand an instance of an application (e.g. HBase)
- cluster.
  
+ Twill and Hoya are therefore very different. One is a platform for new YARN applications,
the other a YARN application to adapt existing applications to YARN.
- What could be of mutual benefit would be to take some of Hoya's logic for
- maintaining a long-lived YARN application and make it re-usable by other applications
- -where clearly Twill would be the ideal destination.
  
+ Although one could argue that Hoya can be written using Twill libraries (which is something
we should pursue as part of long/medium-term collaboration between the two projects), The
goals of the two projects are different - Twill would continue to make YARN application developers'
lives easier, while Hoya is a tool that could deploy distributed-frameworks easily in a YARN
cluster, and be able to later do basic management . Things like dynamic patching of the application's
configuration to run in the YARN cluster, failure detection reacting to failures, storing
some state about applications to facilitate better application restart behavior in a YARN
cluster, etc. would be in the purview of Hoya. Management frameworks could use Hoya as a tool
to start/stop/shrink/expand an instance of an application.
- Hoya  a clean model-view split server side, with all the server state stored independently
- from the AM's entry point and RPC client and service interfaces. This was done
- to isolate the model and aid mock testing of large clusters with simulated scale,
- and hence increase confidence that Hoya can scale to work in large YARN clusters
- and with larger application instances -without mandating that every developer
- has unrestricted access to a 500 node cluster with the SSH credentials necessary
- to trigger remote process failure and hence failure recovery.
  
+ ==== Relationship with Apache Helix ====
- This state is something that could perhaps be worked out into something
- more re-usable, as all it really does is
  
+ Hoya shares some common goals with [[http://helix.incubator.apache.org/|Apache Helix]].
 Helix is more sophisticated and is designed to work standalone. Hoya is designed to work
only in the context of a YARN cluster,  and focuses on that YARN integration.
-  1. Take a specification of role types, the no. of instances of each, and some options (there's
an unused anti-affinity flag)
-  1. Build a model of the cluster, including live containers, queued operations, nodes in
the YARN cluster and their history
-  1. Build up a list of actions: requests and releases aimed at keeping the model (and hence
the deployed app) consistent with the specification.
-  1. Persist placement history to HDFS for best-effort reassignment requests on cluster restart
-  1. Respond to relayed events such as startup, AM restart, container assignment, container
failure, and specification updates by generating revised list of container request/release
operations
  
+ We have discussed Hoya with the Helix team, and feel that the work we are doing in YARN
integration -and driving YARN changes, will be of direct benefit to Helix. We plan to collaborate
on features which can be shared across both projects.
- For any long-lived YARN service, the goal "keep the number of
- instances of a role constant" is an underlying use case, from
- Distributed Shell up. Hoya just effectively allows this to be driven
- by an JSON document persisted to HDFS, updated via RPC, and delegates
- to plugin providers the work of actually starting processes at the far
- end.
  
+ ==== Relationship with Apache Accumulo and Apache HBase ====
- Making this reusable would not only assist anyone else writing
- long-lived services, but there are
- some aspect of failure handling that we can improve in a way
- that would benefit all apps, such as better weighted-moving average
- failure tracking, sophisticated policies on reacting to it, and moving
- beyond a simple works/doesn't work to a more nuanced greylist of
- perceived server reliability.
  
+ We offer these projects the flexibility of operation in a YARN cluster.  As such, it should
expand the uses of the applications, and their user base. 
- Similarly, if we develop a management API that can be used by
- cluster management tools to perform basic monitoring of long-lived
- YARN applications, such an API could be implemented by other applications
- -a feature that does again imply re-usable code.
  
+ There may be some changes that the applications can make to help them live more easily in
a YARN cluster, and to be managed by Hoya. To date, all changes have been limited to supporting
dynamic port allocations and some exit code changes -minor improvements of benefits to all.

  
+ It may be in future that we encounter situations where other changes to the applications
can help them work even better in Hoya-managed deployments. If these arise we would hope to
work with the relevant teams to get the changes adopted --knowing up front that neither of
these project teams would countenance any changes that interfered with classic static application
deployments.
+ 
+ The initial Hoya committer list includes committers for both Accumulo and HBase, who can
maintain cross-project collaboration.
  == Known Risks ==
  
  The biggest risk is getting the critical mass of use needed to build a broad
@@ -225, +111 @@

  significantly aid adoption and governance.
  
  The other risk is YARN not having the complete feature set needed
- for long lived services: restarting, security token renewal, log capture
+ for long lived services: restarting, security token renewal, log-capture
  and other issues. We are working with the YARN developers to address these
  issues, issues shared with other long-lived services on YARN.
  
+ === Orphaned Products ===
+ 
+ Steve Loughran plans to work full time on the project; the others splitting their time between
Hoya and the applications that it already deploys.  Closer integration with these applications
will increase adoption, broaden the developer base and reduce any risks of orphanage. 
+ 
+ === Inexperience with Open Source ===
+ 
+ All of the core developers have long-standing experience in open source, Two of them are
accumulo committers, two HBase committers. Steve Loughran has been a committer on various
ASF projects since 2001, (Ant, Axis), a mentor to Incubated projects, a Hadoop committer since
2008, and full-time developer on HP's openSource SmartFrog project from 2005-2012.
+ 
+ === Homogeneous Developers ===
+ 
+ The current core developers are all from Hortonworks. However, we hope to establish a developer
community that includes users of Hoya and developers on the applications themselves -HBase,
Accumulo, etc)
+ 
+ === Reliance on Salaried Developers ===
+ 
+ Currently, the developers are paid to do work on Hoya. A key goal for the incubation process
will be to broaden the developer base.
+ 
+ === Relationships with Other Apache Products ===
+ 
+ This is covered in the Alignment section
+ 
+ === An Excessive Fascination with the Apache Brand ===
+ 
+ While we respect the reputation of the Apache brand and have no doubts that it will attract
contributors and users, our interest is primarily to give Hoya a solid home as an open source
project with a broad developer base -and to encourage adoption by the related ASF projects.
  == Documentation ==
  
- All Hoya documentation is currently in [[markdown-formatted text files in the source repository|https://github.com/hortonworks/hoya/tree/master/src/site/markdown]];
+ All Hoya documentation is currently in [[https://github.com/hortonworks/hoya/tree/master/src/site/markdown|markdown-formatted
text files in the source repository]];
  they will be delivered as part of the initial source donation.
  
  == Initial Source ==
  
- All initial source can be found at [[https://github.com/hortonworks/hoya]]
+ The initial  source -all ASF-licensed- can be found at [[https://github.com/hortonworks/hoya]]
+ 
+ Hoya is written in Java. Its source tree is entirely self-contained and relies on Apache
Maven as its build system. Alongside the application and HBase and Accumulo providers, it
contains unit, localhost and functional tests, the latter for use with remote clusters.
  
  == Source and IP Submission Plan ==
  
   1. All source will be moved to Apache Infrastructure
-  1. All outstanding issues in our in-house JIRA infrastructure will be replicated into the
Apache JIRA system.
+  2. All outstanding issues in our in-house JIRA infrastructure will be replicated into the
Apache JIRA system.
-  1. We have a currently unused twitter handle {{{@hoyaproject}}} which would be passed to
the PMC.
+  3. We have a currently-unused twitter handle {{{@hoyaproject}}} which would be passed to
the PMC.
  
  == External Dependencies ==
  
  Hoya has no external dependencies except for some Java libraries that are
+ considered ASF-compatible (JUnit, SLF4J, jcommander, groovy), and Apache artifacts : Hadoop,
HBase, Accumulo, Log4J and the transient dependencies of all these artifacts.
- considered ASF-compatible (JUnit, SLF4J, jcommander), and Apache artifacts : Hadoop, HBase,
- Accumulo, Log4J and others.
  
  == Required Resources ==
  
  Mailing Lists
   1. hoya-dev
-  1. hoya-commits
+  2. hoya-commits
-  1. hoya-private
+  3. hoya-private
  
  Infrastructure
   1. Git repository
-  1. Jenkins builds on x86-Linux, ARM-Linux and Windows hooked up to JIRA
-  1. Gerrit would be useful for reviewing, if available.
+  2. JIRA Hoya (HOYA)
+  3. Gerrit for reviewing patches
+ 
+ The existing code includes local host integration tests, so we would like a Jenkins instance
to run them whenever a new patch is submitted.
  
  == Initial Committers ==
  
   1. Steve Loughran (stevel at a.o)
-  1. Billie Rinaldi (billie at a.o)
+  2. Billie Rinaldi (billie at a.o)
-  1. Ted Yu (tedyu at a.o)
+  3. Ted Yu (tedyu at a.o)
-  1. Josh Elser (elserj at a.o)
+  3. Josh Elser (elserj at a.o)
-  1. Devaraj Das (ddas at a.o)
+  4. Devaraj Das (ddas at a.o)
-  1. Larry McCay (lmccay at a.o)
+  5. Larry !McCay (lmccay at a.o)
-  1. Abhishek Kapoor
+  6. Abhishek Kapoor
  
  == Sponsors ==
  
- Champion
+ Champion: Vinod Kumar Vavilapalli
  
  Nominated Mentors
  
   1. Jean-Baptiste Onofré
+  1. Enis Söztutar
+  1. Vinod Kumar Vavilapalli
+  1. Ashutosh Chauhan
+ 
  
  == Sponsoring Entity ==
  
- Incubator vs Hadoop PMC?  HBase PMC
+ Incubator PMC
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message