Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Devaraj Das <ddas@hortonworks.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Subject: [VOTE] Accept HMS as an incubator project
Date: Thu, 25 Aug 2011 13:57:11 -0700
Message-Id: <4BA01D79-5B9E-43A6-96C2-29A272F94259@hortonworks.com>
To: general@incubator.apache.org
Mime-Version: 1.0 (Apple Message framework v1082.1)

Hello everyone,
This is a vote proposing that HMS be accepted as a project in the Apache =
Incubator. HMS is monitoring, administration and lifecycle management =
project for Apache Hadoop clusters. The latest proposal is pasted at the =
end and it could be found in the wiki as well - =
http://wiki.apache.org/incubator/HMSProposal

The related discussion thread is at:

http://www.mail-archive.com/general@incubator.apache.org/msg30354.html

Please cast your votes:
[  ] +1 Accept HMS for incubation
[  ] +0 Indifferent to HMS incubation
[  ] -1 Reject HMS for incubation (and why)

This vote will close 72 hours  from now.
=20
Thanks,
Devaraj.

=
--------------------------------------------------------------------------=
----

HMS Proposal

Abstract

HMS is monitoring, administration and lifecycle management project for =
Apache Hadoop clusters.

Proposal

HMS will simplify the process of deployment, configuration, management =
and monitoring of the collection of Hadoop services and applications =
that compose a Hadoop cluster. The collection of services (Hadoop Stack) =
will include at least HDFS, MapReduce, HBase, Hive, HCatalog, Pig and =
Zookeeper. HMS will be easily configurable to add additional services =
and applications to the stack. Our plan is to support the Hadoop stack =
as a unit of deployment and configuration where only certain pre-tested =
versions of software components are supported to be part of Hadoop =
stack. Administrators can always enable/disable the individual software =
components from the Hadoop stack per their deployment needs.

The main use cases that HMS is trying to address are the following:

	=95 Hadoop stack deployment and upgrades
	=95 Hadoop services configuration & management

	=95 Administration of Hadoop services
		=95 Includes starting and stopping services
		=95 Hadoop system maintenance tasks, such as fsck, =
format, re-balance, and compaction
	=95 User access & quota management on Hadoop clusters

	=95 Easily check and be alerted to failures in Hadoop servers
	=95 Automated discovery of new machines that become available
	=95 Expanding and contracting Hadoop clusters
	=95 Automatic resynchronization to =91desired=92 state (of =
Hadoop stack) to handle faulty nodes
	=95 Handle node burn-ins (stress test nodes using Hadoop before =
deploying them for production use)
	=95 Simple monitoring and management UI
	=95 Dynamic configuration - Hadoop configuration deduced from =
machine attributes (e.g., RAM, CPU, Disk)
	=95 Operational HBase-based (inspired by OpenTSDB) monitoring =
for Hadoop clusters
	=95 Make it possible for administrators to deploy other Hadoop =
related services and client applications
HMS is targeted to administrators responsible for managing Hadoop =
clusters. HMS leverages existing data center management and monitoring =
infrastructure - Nagios, LDAP, Kerberos, etc. All HMS functionality and =
data will be accessible via RESTFUL APIs and command line tools to =
facilitate its integration with existing data center management suites.

For the bare metal provisioning, the cluster admins continue to use =
their existing infrastructure. Provisioning a machine from scratch is =
not in the scope of the current roadmap.

Background

Hadoop=92s ecosystem includes many projects (HDFS, MapReduce, Pig, =
HBase, etc.). In many cases, users and operators typically want to =
deploy a combination of some projects as a stack. It takes a significant =
amount of time to get a properly configured Hadoop cluster up and =
running. HMS has been designed to solve that problem. HMS automates the =
whole process of deploying a stack.

HMS is being developed by developers employed with Yahoo!, Hortonworks =
and IBM. Such a tool would have a large number of users and increase the =
adoption of Apache Hadoop=92s ecosystem. We are therefore proposing to =
make HMS Apache open source.

Rationale

Hadoop clusters are complicated and difficult to deploy and manage. The =
HMS project aims to improve the usability of Apache Hadoop. Doing so =
will democratize Apache Hadoop, growing its community and increasing the =
places Hadoop can be used and the problems it can solve. By developing =
HMS in Apache we hope to gather a diverse community of contributors, =
helping to make sure that HMS is deployable in as many different =
situations as possible. members of the Hadoop development community will =
be able to influence HMS=92s roadmap, and contribute to it. We believe =
having HMS as part of the Apache Hadoop ecosystem will be a great =
benefit to all of Hadoop's users.

Current Status

Prototype available, developed by the list of initial committers.

Meritocracy

Our intent with this incubator proposal is to start building a diverse =
developer community around HMS following the Apache meritocracy model. =
We have wanted to make the project open source and encourage =
contributors from multiple organizations from the start. We plan to =
provide plenty of support to new developers and to quickly recruit those =
who make solid contributions to committer status.

Community

We are happy to report that multiple organizations are already =
represented by initial team. We hope to extend the user and developer =
base further in the future and build a solid open source community =
around HMS.

Core Developers

HMS is currently being developed by four engineers from Hortonworks - =
Eric Yang, Owen O=92Malley, Vitthal (a.k.a Suhas) Gogate and Devaraj =
Das. In addition, a Yahoo! employee, Jagane Sundar, and an IBM employee, =
Kan Zhang, are also involved. Eric, Jagane and Kan are the original =
developers. All the engineers have deep expertise in Hadoop and are =
quite familiar with the Hadoop Ecosystem.

Alignment

The ASF is a natural host for HMS given that it is already the home of =
Hadoop, Pig, HBase, Cassandra, and other emerging cloud software =
projects. HMS has been designed to solve the deployment, management and =
configuration problems of the Hadoop ecosystem family of products. HMS =
fills the gap that Hadoop ecosystem has been lacking in the areas of =
configuration, deployment and manageability.

Known Risks

Orphaned products & Reliance on Salaried Developers

The core developers plan to work full time on the project. There is very =
little risk of HMS getting orphaned. HMS is in use by companies we work =
for so the companies have an interest in its continued vitality.

Inexperience with Open Source

All of the core developers are active users and followers of open =
source. Eric Yang is a committer on Apache Chukwa. Owen O=92Malley is =
the lead of the Apache Hadoop project. Devaraj Das is an Apache Hadoop =
committer and Apache Hadoop PMC member. Vitthal (Suhas) Gogate has =
contributed extensively to the Hadoop Vaidya project (part of Apache =
Hadoop). Jagane Sundar has been contributing, in terms of ideas, to the =
Hadoop project. Kan Zhang is a Hadoop Committer.

Homogeneous Developers

The current core developers are from Hortonworks, IBM, and, Yahoo!. =
However, we hope to establish a developer community that includes =
contributors from several corporations.

Reliance on Salaried Developers

Currently, the developers are paid to do work on HMS. However, once the =
project has a community built around it, we expect to get committers and =
developers from outside the current core developers.

Relationships with Other Apache Products

HMS is going to be used by the users of Hadoop and the Hadoop ecosystem =
in general.

A Excessive Fascination with the Apache Brand

While we respect the reputation of the Apache brand and have no doubts =
that it will attract contributors and users, our interest is primarily =
to give HMS a solid home as an open source project following an =
established development model. We have also given reasons in the =
Rationale and Alignment sections.

Documentation

There is documentation in Hortonworks=92s internal repositories.

Initial Source

The source is currently in Hortonworks=92s internal repositories.

Source and Intellectual Property Submission Plan

The complete HMS code is under Apache Software License 2.

External Dependencies

The dependencies all have Apache compatible licenses. These include BSD, =
MIT licensed dependencies.

Cryptography

None

Required Resources

Mailing lists

	=95 hms-dev AT incubator DOT apache DOT org
	=95 hms-commits AT incubator DOT apache DOT org
	=95 hms-user AT hms incubator apache DOT org
	=95 hms-private AT incubator DOT apache DOT org
Subversion Directory

https://svn.apache.org/repos/asf/incubator/hms

Issue Tracking

JIRA HMS

Initial Committers

	=95 Devaraj Das (ddas AT apache DOT org)
	=95 Vitthal Suhas Gogate (gogate AT apache DOT org)
	=95 Owen O'Malley (omalley AT apache DOT org)
	=95 Jagane Sunder (jagane AT sundar DOT org)
	=95 Eric Yang (eyang AT apache DOT org)
	=95 Kan Zhang (kzhang AT apache DOT org)
Affiliations

	=95 Devaraj Das (Hortonworks)
	=95 Vitthal Suhas Gogate (Hortonworks)
	=95 Owen O'Malley (Hortonworks)
	=95 Jagane Sunder (Yahoo)
	=95 Eric Yang (Hortonworks)
	=95 Kan Zhang (IBM)
	=95 Chris Douglas (Yahoo)
	=95 Arun C Murthy (Hortonworks)
Sponsors

Champion

	=95 Owen O'Malley
Nominated Mentors

	=95 Owen O'Malley
	=95 Arun C Murthy
	=95 Chris Douglas
Sponsoring Entity

Incubator PMC=

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org