Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=zReCAXrNRJY5ps0WMeqR1/8Be5aDQmKmFKRkxYF2jxqjOT+rV5Rc9RcJYVVOBjHZ
Message-ID: <4C3CD27F.60003@yahoo-inc.com>
Date: Tue, 13 Jul 2010 13:54:23 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
 rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HEP proposal
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
In-Reply-To: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Eli,

Thanks for a really good proposal.
Some questions / comments:

On voting
1. Which voting rule?
http://www.apache.org/foundation/glossary.html#ConsensusApproval
http://www.apache.org/foundation/glossary.html#MajorityApproval
I think you mean the MajorityApproval as it does not have veto rule.
So may be it's just clarifying the reference.

2. Who can vote?
Usually PMCs have Binding Votes.
Would be good to have a sentence clarifying this.

3. How long does the vote go?
Usual 3 days may not be enough. One week is reasonable?

4. Discussion on public lists.
A HEP can evolve from a jira, then it should be counted as a public
discussion. I think it makes sense even to continue the discussion
there if so. I propose to add the following:
=========================================
idea is HEP-able by sending mail to the general, or the project-specific lists
+	(including Jiras)
if the scope of the idea is limited to the project.
=========================================

5. How the set of editors is selected?
    "The editors are apointed and removed by the PMC informally, similar to
    how the Apache Board appoints shepherds to projects."
This needs a reference. How does Apache Board appoints shepherds?

6. The level of design details.
I think HEP should have a pretty detailed design. When people vote they
will want to be sure the design can lead to a reasonable implementation.
Should we say "implementation-ready design", rather than
"A high-level explanation of the design."
Or just
"A _detailed_ explanation of the design."

7. Typos:
successuflly, apointed, intial

Thanks,
--Konstantin

On 7/12/2010 12:39 PM, Eli Collins wrote:
> A while back we started discussing on list (http://bit.ly/aFj9Ya) and
> at the contributors meeting (http://bit.ly/aj4Y7I) a more coordinated
> way to describe, socialize and shepherd enhancements to Hadoop.
> Thanks for all the feedback.  Most of it was encouraging so I wrote up
> a draft proposal with specifics to discuss here.  After incorporating
> feedback I'll send out another revision for vote.
>
> Thanks,
> Eli
>
>
> HEP: 1
> Title: HEP Purpose and Guidelines
> Author: Eli Collins
> Status: Draft
>
>
> What is a HEP?
> ==============
>
> HEP stands for Hadoop Enhancement Proposal, and is based on Python's
> PEP (Python Enhancement Proposal) [1].  A HEP is a document that
> describes a new feature, it's rationale, and issues the feature needs
> to address in order to be successuflly incorporated.
>
> The intent is for HEPs to be the primary mechanism for proposing
> significant new features to core Hadoop (common, HDFS and MapReduce),
> incorporating community feedback, and recording the proposal.  Going
> through the HEP process should improve the chances that a proposal is
> successful.
>
> While HEPs do not need to come with code, they are a mechanism to
> propose features to the community, with the intent of contributing the
> feature, rather than request the community implement a feature.
>
> HEPs must be consistent with Apache bylaws [2], for example, the HEP
> workflow takes place on the public Apache Hadoop lists.
>
>
> When is a HEP Required?
> =======================
>
> HEPs should not impede casual contribution to Hadoop.  Small
> improvements and bugs do not require HEPs.  Not all features need
> HEPs.  While the decision is subjective, here are some guidelines to
> indicate a HEP should be considered:
>
> - The feature impacts backwards compatibility (eg modifies released
> public APIs in an incompatible way).
>
> - The feature requires that an existing component be substantially
> re-designed (eg NameNode modified to use Bookkeeper).
>
> - The implementation impact multiple parts of the system (eg symbolic
> links versus adding a pluggable component like a codec).
>
> - The feature impacts the entire development community (eg converts
> the build system to use maven).
>
>
> HEP Workflow
> ============
>
> The author of a HEP should first try to determine if their idea is
> HEP-able by sending mail to the general, or the project-specific lists
> if the scope of the idea is limited to the project.  This gives the
> author a chance to flesh out the proposal, address intial concerns,
> and figure out whether it has a chance of being accepted.  The
> author's role is to build consensus, and gather dissenting opinions.
>
> Following this discussion the author should draft a HEP proposal
> following the HEP template. The proposal should accurately reflect and
> address feedback and dissenting opinions.  For example, flesh out
> sections on backwards compatibility or testing. The author should send
> the draft of the proposal to hep@hadoop.apache.org for review.  This
> is a new, public list for editors and those interested in following
> the review process.
>
> A set of editors reviews incoming HEPs. Each HEP is assigned a single
> primary editor. An editor may volunteer if they feel particular
> functional expertise is required or assign HEPs to editors round
> robin.
>
> The editor reviews the proposal and may request it be updated if it
> does not sufficiently address feedback raised during discussion, eg
> why the proposal is not redundant with existing functionality, or is
> technically sound, sufficiently motivated, covers backwards
> compatibility, etc. As updates are necessary, the HEP author can check
> in new versions if they have commit permissions, or can email new HEP
> versions to the editor for committing. In order to ensure HEP
> proposals make progress the editor should respond to proposal drafts
> within two weeks of receiving them (or the proposer can request
> another editor), and the proposer should generate updates to the draft
> within two weeks of receiving feedback from the editor.
>
> The editor's role is to determine if the proposal is complete, so that
> the proposal can be voted on, not whether they agree with the proposal
> itself.  The editor's involvement should increase the chance that a
> HEP proposal makes it to a vote.
>
> Once the editor deems the proposal is complete they add it to a
> versioned HEP repository and the author posts the proposal to
> general@hadoop.apache.org for vote.  HEP votes, like Apache procedural
> votes, use majority rule [3]. Successful HEPs are assigned a number,
> unsuccessful HEPs remain drafts.
>
> The editors are apointed and removed by the PMC informally, similar to
> how the Apache Board appoints shepherds to projects.
>
>
> HEP Contents
> ============
>
> Each HEP should contain the following:
>
> 1. Preamble -- Including the HEP number, a short descriptive title,
> and the names of the authors.
>
> 2. Abstract -- A short (~200 word) description of the technical issue
> being addressed.
>
> 3. Copyright/public domain -- Each HEP must either be explicitly
> labelled as placed in the public domain (see this HEP as an example).
>
> 4. Design -- A high-level explanation of the design. It should cover
> intended use cases, failure scenarios, and impact on the existing
> system.
>
> 5. Motivation -- The motivation spells out the use case for the
> feature and the benefits it provides.
>
> 6. Rationale -- The rationale describes what motivated the design and
> why particular design decisions were made.  It should describe
> alternate designs that were considered and related work, e.g. how the
> feature is designed in other systems. It should also consider whether
> the feature could be achieved by layering atop the existing system
> rather than modifying it.
>
> The rationale should provide evidence of consensus within the
> community and discuss important objections or concerns raised during
> discussion.
>
> 7. Backwards Compatibility -- All HEPs that introduce backwards
> incompatibilities must include a section describing these
> incompatibilities and their severity.  The HEP must explain how the
> author proposes to deal with these incompatibilities.  HEP submissions
> without a sufficient backwards compatibility treatise may be rejected
> outright.
>
>
> HEP Template
> ============
>
> HEPs should be plain text with minimal structural markup that adheres
> to a rigid style.  You can use this HEP as an example. Each HEP starts
> with a header that contains the HEP number (or empty if the number has
> not yet been assigned), title, list of authors and status (Draft,
> Accepted, Rejected, or Withdrawn).
>
>
> Auxiliary Files
> ===============
>
> HEPs may include auxiliary files such as diagrams.  Such files must be
> named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is a
> serial number (starting at 1), and "ext" is replaced by the actual
> file extension (e.g. "png").
>
>
> References
> ==========
>
> 1. http://www.python.org/dev/peps/pep-0001
>
> 2. http://www.apache.org/foundation/bylaws.html
>
> 3. http://www.apache.org/foundation/voting.html
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>