incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "HMSProposal" by DevarajDas
Date Fri, 19 Aug 2011 18:38:46 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "HMSProposal" page has been changed by DevarajDas:
http://wiki.apache.org/incubator/HMSProposal?action=diff&rev1=12&rev2=13

  
  == Proposal ==
  
- We are proposing an Apache project to build a tool that would vastly simplify the process
of deployment and configuration of Hadoop stack on a cluster. Hadoop stack comprises of various
software components in the Hadoop ecosystem e.g. HDFS, MapReduce, HBase, Hive, HCatalog, Pig,
Zookeeper and Oozie. Our plan is to support the Hadoop stack as a unit of deployment and configuration
where only certain pre-tested versions of software components are supported to be part of
Hadoop stack. Administrators can always enable/disable the individual software components
from the Hadoop stack per their deployment needs.
+ HMS will simplify the process of deployment, configuration, management and monitoring of
the collection of Hadoop services and applications that compose a Hadoop cluster. The collection
of services (Hadoop Stack) will include at least HDFS, MapReduce, HBase, Hive, HCatalog, Pig
and Zookeeper. HMS will be easily configurable to add additional services and applications
to the stack. Our plan is to support the Hadoop stack as a unit of deployment and configuration
where only certain pre-tested versions of software components are supported to be part of
Hadoop stack. Administrators can always enable/disable the individual software components
from the Hadoop stack per their deployment needs.
  
  The main use cases that HMS is trying to address are the following:
   * Hadoop stack deployment and upgrades 
   * Hadoop services configuration & management
-   * Declarative configuration (no scripts required)
   * Administration of Hadoop services
    * Includes starting and stopping services
-   * System maintenance tasks, such as fsck, format, re-balance, and compaction
+   * Hadoop system maintenance tasks, such as fsck, format, re-balance, and compaction
   * User access & quota management on Hadoop clusters
   * Easily check and be alerted to failures in Hadoop servers
   * Automated discovery of new machines that become available
   * Expanding and contracting Hadoop clusters
-  * Automatic resynchronization to ‘desired’ state to handle faulty nodes
+  * Automatic resynchronization to ‘desired’ state (of Hadoop stack) to handle faulty
nodes
-  * Handle node burn-ins
-  * In the future, possibly allow for customized monitoring dashboards
+  * Handle node burn-ins (stress test nodes using Hadoop before considering them for production
use)
+  * Simple monitoring and management UI
   * Dynamic configuration - Hadoop configuration deduced from machine attributes (e.g., RAM,
CPU, Disk)
-  * Operational monitoring for Hadoop clusters
+  * Operational HBase-based (inspired by OpenTSDB) monitoring for Hadoop clusters
+  * Make it possible for administrators to deploy other Hadoop related services and client
applications
  
- HMS is targeted to administrators responsible for managing Hadoop clusters. HMS leverages
existing data center management and monitoring infrastructure - Nagios, LDAP, Kerberos, etc.
+ HMS is targeted to administrators responsible for managing Hadoop clusters. HMS leverages
existing data center management and monitoring infrastructure - Nagios, LDAP, Kerberos, etc.
All HMS functionality and data will be accessible via RESTFUL APIs and command line tools
to facilitate its integration with existing data center management suites.
  
  For the bare metal provisioning, the cluster admins continue to use their  existing infrastructure.
Provisioning a machine from scratch is not in the scope of the current roadmap.
  
@@ -39, +39 @@

  
  == Rationale ==
  
- The reasons for having a tool like HMS have been explained above. Having HMS as an Apache
Open Source project will highly benefit it from the point of view of getting a large community
that currently uses Hadoop and the other products built around Hadoop (like Pig, Hive, etc.).
Users of the Hadoop ecosystem can influence HMS’s roadmap, and contribute to it. Looking
at it in another way, we believe having HMS as part of the Hadoop ecosystem will be a great
benefit to the current Hadoop ecosystem too.
+ Hadoop clusters are complicated and difficult to deploy and manage.  The HMS project aims
to improve the usability of Apache Hadoop.  Doing so will demoncratize Apache Hadoop, growing
its community and increasing the places Hadoop can be used and the problems it can solve.
  By developing HMS in Apache we hope to gather a diverse community of contributors, helping
to make sure that HMS is deployable in as many different situations as possible.  members
of the Hadoop development community will be able to influence HMS’s roadmap, and contribute
to it.   We believe having HMS as part of the Apache Hadoop ecosystem will be a great benefit
to all of Hadoop's users.
  
  == Current Status ==
+ 
+ Prototype available, developed by the list of initial committers.
  
  === Meritocracy ===
  
@@ -49, +51 @@

  
  === Community ===
  
- HMS is currently being worked on by developers from Hortonworks and there has been an expressed
interest from people at Yahoo!. There are users within Hortonworks & Yahoo! who use the
existing prototype for doing deployments of the Hadoop stack in lab environments. We hope
to extend the user and developer base further in the future and build a solid open source
community around HMS.
+ We are happy to report that multiple organizations are already represented by initial team.
 We hope to extend the user and developer base further in the future and build a solid open
source community around HMS.
  
  === Core Developers ===
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message