incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: [VOTE] Heron to enter Apache Incubator
Date Sat, 17 Jun 2017 03:58:05 GMT
+1 (binding)

> On Jun 16, 2017, at 8:43 PM, Nabarun Nag <nnag@apache.org> wrote:
> 
> +1
> 
> On Fri, Jun 16, 2017 at 8:28 PM Jacques Nadeau <jacques@apache.org> wrote:
> 
>> +1
>> 
>> On Jun 16, 2017 7:26 PM, "Ashish" <paliwalashish@gmail.com> wrote:
>> 
>> +1 (non-binding)
>> 
>> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <billgraham@gmail.com> wrote:
>>> Hi,
>>> 
>>> Based on the discussion on the incubator mailing list[1] I would like to
>>> call a vote to add Heron to the Apache Incubator.
>>> 
>>> The full proposal is available below, and is also available on the Apache
>>> Incubator wiki at:
>>>    https://wiki.apache.org/incubator/HeronProposal
>>> 
>>> Please vote:
>>>  [ ] +1, bring Heron into Incubator
>>>  [ ] -1, do not bring Heron into Incubator, because...
>>> 
>>> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>>> 
>>> Thank you
>>> 
>>> 1 -
>>> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
>> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>>> 
>>> 
>>> 
>>> = Heron Proposal =
>>> 
>>> = Abstract =
>>> Heron is a real-time, distributed, fault-tolerant stream processing
>> engine
>>> initially developed by Twitter.
>>> 
>>> = Proposal =
>>> 
>>> Heron is a real-time stream processing engine built for high performance,
>>> ease of manageability, performance predictability and developer
>>> productivity[1]. We wish to develop a community around Heron to increase
>>> contributions and see Heron thrive in an open forum.
>>> 
>>> = Background =
>>> 
>>> Heron provides the ability for developers to compose directed acyclic
>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>>> submit the topology to execute on a pluggable job scheduling system
>> (e.g.,
>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>>> Heron API or the Apache Storm API to develop the topology. Heron supports
>>> the Storm API for ease of migration, but beyond that Heron’s architecture
>>> differs considerably from Storm’s.
>>> 
>>> Users submit a topology to the scheduler using the Heron client, which
>> uses
>>> the Heron binary libraries to deploy all daemons required to run and
>> manage
>>> the topology. The topology therefore has no reliance on centrally managed
>>> Heron services, only on a generic job scheduling system, which lends
>> itself
>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
>>> others).
>>> 
>>> The scheduler runs each topology as a job consisting of multiple
>>> containers. One of the containers runs the topology master, responsible
>> for
>>> managing the topology. The remaining containers each runs a stream
>> manager
>>> responsible for data routing, a metrics manager that collects and reports
>>> various metrics and a number of processes called Heron instances which
>> run
>>> the user-defined logic on the stream of tuples. Parallelism is achieved
>> via
>>> process-based isolation of Heron instances, which provides predictable
>>> performance while simplifying debugging. The containers are allocated and
>>> managed by the scheduler framework based on resource availability of
>> nodes
>>> in the cluster. The metadata for the topology, such as the physical plan
>>> and execution details, are stored in the pluggable Heron State Manager
>>> (e.g. Apache ZooKeeper).
>>> 
>>> = Rationale =
>>> 
>>> Heron is a general-purpose, modular and extensible platform that can be
>>> leveraged to support common, real-time analytics use cases. There is an
>>> increasing demand for open-source, scalable real-time analytics systems.
>> We
>>> believe that Heron can be leveraged by other organizations to build
>>> streaming applications that can benefit from its robustness, high
>>> performance, adaptability to cloud environments and ease of use.
>> Moreover,
>>> we hope that open-sourcing Heron will help to further evolve the
>> technology
>>> as the project attracts contributors with diverse backgrounds and areas
>> of
>>> expertise.
>>> 
>>> We believe the Apache foundation is a great fit as the long-term home for
>>> Heron, as it provides an established process for community-driven
>>> development and decision making by consensus. This is exactly the model
>> we
>>> want for future Heron development.
>>> 
>>> = Initial Goals =
>>> 
>>> * Move the existing codebase, website, documentation, and mailing lists
>> to
>>> Apache-hosted infrastructure.
>>> * Integrate with the Apache development process.
>>> * Ensure all dependencies are compliant with Apache License version 2.0.
>>> * Incrementally develop and release per Apache guidelines.
>>> 
>>> = Current Status =
>>> 
>>> Heron is a stable project used in production at Twitter since 2014 and
>> open
>>> sourced under the ASL v2 license in 2016. The Heron source code is
>>> currently hosted at github.com (https://github.com/twitter/heron), which
>>> will seed the Apache git repository.
>>> 
>>> = Meritocracy =
>>> 
>>> By submitting this incubator proposal, we’re expressing our intent to
>> build
>>> a diverse developer community around Heron that will conduct itself
>>> according to The Apache Way and use a meritocratic means of building it's
>>> committer base. Several companies and universities have already expressed
>>> interest in and contributed to Heron. Our goal is to grow the Heron
>>> community by encouraging open communication, contribution and
>> participation
>>> of all types, and ensuring that contributors are recognized
>> appropriately.
>>> 
>>> = Community =
>>> 
>>> Heron is currently being used by Twitter, Google, Machine Zone and
>>> ndustrial.io and has received significant contributions by Microsoft and
>>> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
>>> attract even more developers who are interested in creating real-time
>>> systems to build the project's contributor base.
>>> 
>>> == Core Developers ==
>>> 
>>> Current core developers are engineers from Twitter, Google, Microsoft and
>>> Streamlio.
>>> 
>>> == Alignment ==
>>> 
>>> Heron utilizes a number of Apache technologies. Heron leverages Apache
>>> ZooKeeper for coordination and has scheduler implementations to integrate
>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>> REEF)
>>> as well as spout implementations to integrate with Apache Kafka and
>> metrics
>>> implementations to integrate with Scribe. Heron also implements the
>> Apache
>>> Storm user-level API, which allows topologies written against Storm to
>> run
>>> in Heron. We believe that having Heron at Apache will help further the
>>> growth of the streaming compute community, as well as encourage
>> cooperation
>>> and developer cross pollination with other Apache projects.
>>> 
>>> = Known Risks =
>>> 
>>> == Orphaned Products ==
>>> 
>>> The risk of the Heron project being abandoned is minimal. It is used in
>>> production at Twitter and Google and other companies are evaluating or
>>> adopting it for production use.
>>> 
>>> == Inexperience with Open Source ==
>>> 
>>> All of the core contributors to the project have considerable experience
>>> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
>>> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
>>> Apache projects and Bill and Ashvin have gone through the Apache
>> incubator
>>> process. Twitter has already donated numerous projects to the ASF (e.g.,
>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
>>> by experienced ASF members that can help with any roadblocks.
>>> 
>>> == Homogenous Developers ==
>>> 
>>> Initial committers come from 5 separate organizations. Our intention is
>>> increase the diversity of contributing developers and their affiliations.
>>> To date github contributions have come from approximately 50 contributors
>>> from outside the Twitter team.
>>> 
>>> == Reliance on Salaried Developers ==
>>> 
>>> It is expected that Heron development will occur on both salaried time
>> and
>>> on volunteer time. The majority of initial committers are paid by their
>>> employers to contribute to this project. We are committed to recruiting
>>> additional committers from other organizations as well as non-salaried
>>> committers to join project.
>>> 
>>> == Relationships with Other Apache Products ==
>>> 
>>> As mentioned in the Alignment section, Heron implements the Apache Storm
>>> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>>> Thrift.
>>> 
>>> == An Excessive Fascination with the Apache Brand ==
>>> 
>>> Heron's popularity is growing in the streaming compute space and we are
>>> long time supporters of the Apache brand. This proposal is not for the
>>> purpose of generating publicity through. Rather, the primary benefits to
>>> joining Apache are those of community building and open decision making
>>> outlined in the Rationale section.
>>> 
>>> == Documentation ==
>>> 
>>> This proposal exists online as
>>> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
>> can
>>> be found on github at https://twitter.github.io/heron and the source
>> code
>>> is well documented.
>>> 
>>> == Source and Intellectual Property Submission Plan ==
>>> 
>>> The Heron codebase is currently hosted on Github:
>>> https://github.com/twitter/heron. During incubation, the codebase will
>> be
>>> migrated to Apache infrastructure. The source code is already ASF 2.0
>>> licensed.
>>> 
>>> == External Dependencies ==
>>> 
>>> All external libraries have ASF 2.0 compatible licenses except for
>> pylint.
>>> The pylint library is GPL licensed, but is only used for pre-build Python
>>> style checks and is neither bundled with, nor relied upon by, the Heron
>>> source or binary release artifacts.
>>> 
>>> == Cryptography ==
>>> 
>>> Heron does not use any cryptography libraries.
>>> 
>>> = Required Resources =
>>> 
>>> == Mailing lists ==
>>> 
>>> * private@heron.incubator.apache.org (with moderated subscriptions)
>>> * dev@heron.incubator.apache.org
>>> * commits@heron.incubator.apache.org
>>> * user@heron.incubator.apache.org
>>> 
>>> == Subversion Directory ==
>>> 
>>> Git is the preferred source control system: git://git.apache.org/heron
>>> 
>>> == Issue Tracking ==
>>> 
>>> JIRA: Heron (HERON)
>>> 
>>> == Initial Committers ==
>>> 
>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>>> * Ashvin Agrawal (ashvin at apache dot org)*
>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>>> * Bill Graham (billgraham at apache dot org)*
>>> * Brian Hatfield (bmhatfield at gmail dot com)
>>> * Chris Kellogg (cckellogg at gmail dot com)
>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>>> * Karthik Ramasamy (karthik at gmail dot com)
>>> * Maosong Fu (maosongfu at gmail dot com)
>>> * Neng Lu(freeneng at gmail dot com)
>>> * Runhang Li (obj dot runhang at gmail dot com)
>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>>> * Supun Kamburugamuve (supun at apache dot org)*
>>> * Thomas Sun (tom dot ssf at gmail dot com)
>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>>> 
>>> == Affiliations ==
>>> 
>>> * Andrew Jorgensen (Google)
>>> * Ashvin Agrawal (Microsoft)
>>> * Avrilia Floratou (Microsoft)
>>> * Bill Graham (Twitter)
>>> * Brian Hatfield (Google)
>>> * Chris Kellogg (Twitter)
>>> * Huijun Wu (Twitter)
>>> * Karthik Ramasamy (Streamlio)
>>> * Maosong Fu (Twitter)
>>> * Neng Lu (Twitter)
>>> * Runhang Li (Twitter)
>>> * Sanjeev Kulkarni (Streamlio)
>>> * Supun Kamburugamuve (Indiana University)
>>> * Thomas Sun (Twitter)
>>> * Yaliang Wang (Twitter)
>>> 
>>> = Sponsors =
>>> 
>>> == Champion ==
>>> 
>>> * Julien Le Dem (julien at apache dot org)
>>> 
>>> == Nominated Mentors ==
>>> 
>>> * Jake Farrell (jfarrell at apache dot org)
>>> * Jacques Nadeau (jacques at apache dot org)
>>> * Julien Le Dem (julien at apache dot org)
>>> * P. Taylor Goetz (ptgoetz at apache dot org)
>>> 
>>> == Sponsoring Entity ==
>>> 
>>> The Apache Incubator
>>> 
>>> == Footnotes ==
>>> 
>>> * 1 - Papers detailing Heron are available at
>>> http://dl.acm.org/citation.cfm?id=2742788 and
>>> http://sites.computer.org/debull/A15dec/p15.pdf.
>>> * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>>> * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>>> * 4 - http://home.apache.org/phonebook.html?uid=supun
>> 
>> 
>> 
>> --
>> thanks
>> ashish
>> 
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message