Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOqetn-4PzHPXz3tSiCaZLsHx-OL6xJ27AQWcZo1UcwXmfB-9A@mail.gmail.com>
References: <CAB-acjPajJUVZ7wC6WyP2SwTpjiO8rH9t9wOh80869+iUA5Dtw@mail.gmail.com>
 <0B5EFEE4-0F9F-475C-8A1A-1C3433858122@gmail.com> <CAB-acjP-pOJhLLC=4M1M4Uv_c-guwxaecy6eaQ-HfFpx5pXwFQ@mail.gmail.com>
 <CAENxBwyBOnER4MFST6s06F6GqX9fCSDvmkDAeGtkysQARDBxjA@mail.gmail.com>
 <7B86462C-770A-43F0-8299-3FEE11036D29@apache.org> <CAB-acjMXmkEnsqyfuRDEA=EOYr88jDG--O8w_bvq_Lcb8a3dKA@mail.gmail.com>
 <CAOqetn-4PzHPXz3tSiCaZLsHx-OL6xJ27AQWcZo1UcwXmfB-9A@mail.gmail.com>
From: Edward Capriolo <edlinuxguru@gmail.com>
Date: Fri, 23 Jun 2017 09:54:57 -0400
Message-ID: <CAENxBwzDSWb9Z6e-Usy=zMYzDW4tVnqyr+Pe9q-Ldg74kVb-iQ@mail.gmail.com>
Subject: Re: [VOTE] Heron to enter Apache Incubator
To: general@incubator.apache.org
Content-Type: multipart/alternative; boundary="94eb2c1cf1ba65391e0552a0f248"
archived-at: Fri, 23 Jun 2017 13:55:12 -0000

--94eb2c1cf1ba65391e0552a0f248
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

"The only overlap is that Heron supports the Storm user API for ease of
migration."

It sounds possible possible that storm could be one user facing API with
two back ends inside one project.

"Accumulo vs HBase" I do not think Accumulo and HBase is a valid comparison
one did not start out to emulate or be compatible with the other.

In any case the largest issue I see is community. The proposed Heron
committer list is mostly a single company. Storm has already established a
community
with diverse committers. Also in terms of adoption, suppose you are a Storm
user do you run Heron side by side? Suppose your a vendor that packages
Hadoop and friends do you ship
both? Suppose you provide a no-sql database do you manage (test, document)
a connector for Heron and Storm? In my experience it is not trivial to keep
something working for example AbcBolt across Storm versions now that matrix
would double.

I wish their was a stronger statement of cooperation in the proposal, for
example, "We with to establish a middle ground repo with shared components
etc". If nothing is shared other than a mentor or PMC your could run
into "software
X is the fasted way to run your storm bolts and spouts because of our
special sauce software Y does not have" and "software x is 2.2 years behind
the api of software Y they only implement and test 10% of the spouts we
support"


On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <johndament@apache.org>
wrote:

> Bill,
>
> Would I be correct in understanding that Heron implements the same protoc=
ol
> as Storm, but the actual implementation is different?
>
> John
>
> On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <billgraham@gmail.com> wrote:
>
> > It's grossly inaccurate to refer to Heron as a Storm fork. There are
> about
> > 132k lines of code in the Heron codebase (plus 166k of codegen), of whi=
ch
> > about 7k are to implement the Apache Storm API bindings to the Heron AP=
I.
> >
> > The Rationale section of the proposal discusses the Heron architecture,
> > which is a complete rewrite with little in common with Storm. The only
> > overlap is that Heron supports the Storm user API for ease of migration=
.
> >
> > The value of having multiple projects to solve a common need is that ea=
ch
> > can foster innovation, collaboration and exchange of ideas in different
> > ways. This is not a new concept to Apache. You can look at the incubato=
r
> > discussions around Accumulo vs HBase (two implementations of the BigTab=
le
> > paper) for example, to see how two different approaches to a shared
> problem
> > can be a good thing.
> >
> > thanks,
> > Bill
> >
> > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vongosling@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I will give +1(Non-binding), but,
> > >
> > > I have the similar question about so many streaming framework in the
> > > apache, how to develop community for themselves.
> > >
> > >
> > >
> > >
> > > Best Regards,
> > > Von Gosling
> > >
> > >
> > >
> > > =E5=9C=A8 2017=E5=B9=B46=E6=9C=8823=E6=97=A5=EF=BC=8C08:51=EF=BC=8CEd=
ward Capriolo <edlinuxguru@gmail.com> =E5=86=99=E9=81=93=EF=BC=9A
> > >
> > > I believe heron and storm should be merged back together. I do not se=
e
> > the
> > > value of storm and a storm fork in the asf.
> > >
> > > On Thursday, June 22, 2017, Bill Graham <billgraham@gmail.com> wrote:
> > >
> > > Thanks Taylor for relaying these sentiments, especially the part abou=
t
> > the
> > > Heron website which is indeed poorly worded (I suspect this could hav=
e
> > been
> > > the result of internal docs being open-sourced). I've opened this pul=
l
> > > request to update the language regarding Storm:
> > >
> > > https://github.com/twitter/heron/pull/1979
> > >
> > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > The Apache Storm PMC had a discussion regarding the Heron proposal. I=
n
> > >
> > > the
> > >
> > > spirit of openness I wanted to bring some of the sentiments expressed
> in
> > > that discussion back to this list. Please note that I am paraphrasing
> > >
> > > from
> > >
> > > that discussion and attempting to relay opinions of the collective PM=
C,
> > >
> > > not
> > >
> > > necessarily that of any individual.
> > >
> > > * There is a general disappointment that the Heron community chose no=
t
> to
> > > engage with the Storm community and instead chose a separate path.
> > > * A majority of the PMC supports Heron=E2=80=99s incubation, though s=
ome felt
> it
> > > would result in unnecessary duplication of effort.
> > > * A majority of the PMC supports the two projects working closely
> > > together. A number of PMC members suggested the two projects merge in
> > >
> > > some
> > >
> > > way.
> > > * Many PMC members took issue some of the marketing language on the
> Heron
> > > website, particularly Heron being billed as =E2=80=9Cthe direct succe=
ssor to
> > >
> > > Apache
> > >
> > > Storm=E2=80=9D and the prominent =E2=80=9CUpgrade from Storm=E2=80=9D=
 links.  The main concern
> > >
> > > here
> > >
> > > was such phrasing has somewhat of a hostile tone and undermines the
> > >
> > > desire
> > >
> > > for better collaboration, as well as confusing users.
> > >
> > > One of my goals as a proposed mentor for Heron and a Storm PMC member
> is
> > > to address some of these concerns and encourage collaboration. As I
> > > mentioned to the Storm PMC on that thread, if there are ongoing
> concerns
> > > from either the Storm PMC or the Heron PPMC about me acting as a
> mentor,
> > >
> > > I
> > >
> > > would be willing to step down.
> > >
> > > +1 (binding)
> > >
> > > -Taylor
> > >
> > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > >
> > > <javascript:;>> wrote:
> > >
> > >
> > > Hi,
> > >
> > > Based on the discussion on the incubator mailing list[1] I would like
> > >
> > > to
> > >
> > > call a vote to add Heron to the Apache Incubator.
> > >
> > > The full proposal is available below, and is also available on the
> > >
> > > Apache
> > >
> > > Incubator wiki at:
> > >   https://wiki.apache.org/incubator/HeronProposal
> > >
> > > Please vote:
> > > [ ] +1, bring Heron into Incubator
> > > [ ] -1, do not bring Heron into Incubator, because...
> > >
> > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > >
> > > Thank you
> > >
> > > 1 -
> > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > >
> > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > >
> > >
> > >
> > >
> > > =3D Heron Proposal =3D
> > >
> > > =3D Abstract =3D
> > > Heron is a real-time, distributed, fault-tolerant stream processing
> > >
> > > engine
> > >
> > > initially developed by Twitter.
> > >
> > > =3D Proposal =3D
> > >
> > > Heron is a real-time stream processing engine built for high
> > >
> > > performance,
> > >
> > > ease of manageability, performance predictability and developer
> > > productivity[1]. We wish to develop a community around Heron to
> > >
> > > increase
> > >
> > > contributions and see Heron thrive in an open forum.
> > >
> > > =3D Background =3D
> > >
> > > Heron provides the ability for developers to compose directed acyclic
> > > graphs (DAGs) of real-time query execution logic (i.e. a topology) an=
d
> > > submit the topology to execute on a pluggable job scheduling system
> > >
> > > (e.g.,
> > >
> > > Apache Aurora, YARN, Marathon, etc). Users can employ either the nati=
ve
> > > Heron API or the Apache Storm API to develop the topology. Heron
> > >
> > > supports
> > >
> > > the Storm API for ease of migration, but beyond that Heron=E2=80=99s
> > >
> > > architecture
> > >
> > > differs considerably from Storm=E2=80=99s.
> > >
> > > Users submit a topology to the scheduler using the Heron client, whic=
h
> > >
> > > uses
> > >
> > > the Heron binary libraries to deploy all daemons required to run and
> > >
> > > manage
> > >
> > > the topology. The topology therefore has no reliance on centrally
> > >
> > > managed
> > >
> > > Heron services, only on a generic job scheduling system, which lends
> > >
> > > itself
> > >
> > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > >
> > > (among
> > >
> > > others).
> > >
> > > The scheduler runs each topology as a job consisting of multiple
> > > containers. One of the containers runs the topology master, responsib=
le
> > >
> > > for
> > >
> > > managing the topology. The remaining containers each runs a stream
> > >
> > > manager
> > >
> > > responsible for data routing, a metrics manager that collects and
> > >
> > > reports
> > >
> > > various metrics and a number of processes called Heron instances whic=
h
> > >
> > > run
> > >
> > > the user-defined logic on the stream of tuples. Parallelism is achiev=
ed
> > >
> > > via
> > >
> > > process-based isolation of Heron instances, which provides predictabl=
e
> > > performance while simplifying debugging. The containers are allocated
> > >
> > > and
> > >
> > > managed by the scheduler framework based on resource availability of
> > >
> > > nodes
> > >
> > > in the cluster. The metadata for the topology, such as the physical
> > >
> > > plan
> > >
> > > and execution details, are stored in the pluggable Heron State Manage=
r
> > > (e.g. Apache ZooKeeper).
> > >
> > > =3D Rationale =3D
> > >
> > > Heron is a general-purpose, modular and extensible platform that can =
be
> > > leveraged to support common, real-time analytics use cases. There is =
an
> > > increasing demand for open-source, scalable real-time analytics
> > >
> > > systems.
> > >
> > > We
> > >
> > > believe that Heron can be leveraged by other organizations to build
> > > streaming applications that can benefit from its robustness, high
> > > performance, adaptability to cloud environments and ease of use.
> > >
> > > Moreover,
> > >
> > > we hope that open-sourcing Heron will help to further evolve the
> > >
> > > technology
> > >
> > > as the project attracts contributors with diverse backgrounds and are=
as
> > >
> > > of
> > >
> > > expertise.
> > >
> > > We believe the Apache foundation is a great fit as the long-term home
> > >
> > > for
> > >
> > > Heron, as it provides an established process for community-driven
> > > development and decision making by consensus. This is exactly the mod=
el
> > >
> > > we
> > >
> > > want for future Heron development.
> > >
> > > =3D Initial Goals =3D
> > >
> > > * Move the existing codebase, website, documentation, and mailing lis=
ts
> > >
> > > to
> > >
> > > Apache-hosted infrastructure.
> > > * Integrate with the Apache development process.
> > > * Ensure all dependencies are compliant with Apache License version
> > >
> > > 2.0.
> > >
> > > * Incrementally develop and release per Apache guidelines.
> > >
> > > =3D Current Status =3D
> > >
> > > Heron is a stable project used in production at Twitter since 2014 an=
d
> > >
> > > open
> > >
> > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > currently hosted at github.com (https://github.com/twitter/heron),
> > >
> > > which
> > >
> > > will seed the Apache git repository.
> > >
> > > =3D Meritocracy =3D
> > >
> > > By submitting this incubator proposal, we=E2=80=99re expressing our i=
ntent to
> > >
> > > build
> > >
> > > a diverse developer community around Heron that will conduct itself
> > > according to The Apache Way and use a meritocratic means of building
> > >
> > > it's
> > >
> > > committer base. Several companies and universities have already
> > >
> > > expressed
> > >
> > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > community by encouraging open communication, contribution and
> > >
> > > participation
> > >
> > > of all types, and ensuring that contributors are recognized
> > >
> > > appropriately.
> > >
> > >
> > > =3D Community =3D
> > >
> > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > ndustrial.io and has received significant contributions by Microsoft
> > >
> > > and
> > >
> > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> > >
> > > can
> > >
> > > attract even more developers who are interested in creating real-time
> > > systems to build the project's contributor base.
> > >
> > > =3D=3D Core Developers =3D=3D
> > >
> > > Current core developers are engineers from Twitter, Google, Microsoft
> > >
> > > and
> > >
> > > Streamlio.
> > >
> > > =3D=3D Alignment =3D=3D
> > >
> > > Heron utilizes a number of Apache technologies. Heron leverages Apach=
e
> > > ZooKeeper for coordination and has scheduler implementations to
> > >
> > > integrate
> > >
> > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > >
> > > REEF)
> > >
> > > as well as spout implementations to integrate with Apache Kafka and
> > >
> > > metrics
> > >
> > > implementations to integrate with Scribe. Heron also implements the
> > >
> > > Apache
> > >
> > > Storm user-level API, which allows topologies written against Storm t=
o
> > >
> > > run
> > >
> > > in Heron. We believe that having Heron at Apache will help further th=
e
> > > growth of the streaming compute community, as well as encourage
> > >
> > > cooperation
> > >
> > > and developer cross pollination with other Apache projects.
> > >
> > > =3D Known Risks =3D
> > >
> > > =3D=3D Orphaned Products =3D=3D
> > >
> > > The risk of the Heron project being abandoned is minimal. It is used =
in
> > > production at Twitter and Google and other companies are evaluating o=
r
> > > adopting it for production use.
> > >
> > > =3D=3D Inexperience with Open Source =3D=3D
> > >
> > > All of the core contributors to the project have considerable
> > >
> > > experience
> > >
> > > with open source software development. Bill Graham[2], Ashvin
> > >
> > > Agrawal[3]
> > >
> > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > >
> > > other
> > >
> > > Apache projects and Bill and Ashvin have gone through the Apache
> > >
> > > incubator
> > >
> > > process. Twitter has already donated numerous projects to the ASF
> > >
> > > (e.g.,
> > >
> > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > >
> > > mentored
> > >
> > > by experienced ASF members that can help with any roadblocks.
> > >
> > > =3D=3D Homogenous Developers =3D=3D
> > >
> > > Initial committers come from 5 separate organizations. Our intention =
is
> > > increase the diversity of contributing developers and their
> > >
> > > affiliations.
> > >
> > > To date github contributions have come from approximately 50
> > >
> > > contributors
> > >
> > > from outside the Twitter team.
> > >
> > > =3D=3D Reliance on Salaried Developers =3D=3D
> > >
> > > It is expected that Heron development will occur on both salaried tim=
e
> > >
> > > and
> > >
> > > on volunteer time. The majority of initial committers are paid by the=
ir
> > > employers to contribute to this project. We are committed to recruiti=
ng
> > > additional committers from other organizations as well as non-salarie=
d
> > > committers to join project.
> > >
> > > =3D=3D Relationships with Other Apache Products =3D=3D
> > >
> > > As mentioned in the Alignment section, Heron implements the Apache
> > >
> > > Storm
> > >
> > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > >
> > > Apache
> > >
> > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apac=
he
> > > Thrift.
> > >
> > > =3D=3D An Excessive Fascination with the Apache Brand =3D=3D
> > >
> > > Heron's popularity is growing in the streaming compute space and we a=
re
> > > long time supporters of the Apache brand. This proposal is not for th=
e
> > > purpose of generating publicity through. Rather, the primary benefits
> > >
> > > to
> > >
> > > joining Apache are those of community building and open decision maki=
ng
> > > outlined in the Rationale section.
> > >
> > > =3D=3D Documentation =3D=3D
> > >
> > > This proposal exists online as
> > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > >
> > > documentation
> > >
> > > can
> > >
> > > be found on github at https://twitter.github.io/heron and the source
> > >
> > > code
> > >
> > > is well documented.
> > >
> > > =3D=3D Source and Intellectual Property Submission Plan =3D=3D
> > >
> > > The Heron codebase is currently hosted on Github:
> > > https://github.com/twitter/heron. During incubation, the codebase wil=
l
> > >
> > > be
> > >
> > > migrated to Apache infrastructure. The source code is already ASF 2.0
> > > licensed.
> > >
> > > =3D=3D External Dependencies =3D=3D
> > >
> > > All external libraries have ASF 2.0 compatible licenses except for
> > >
> > > pylint.
> > >
> > > The pylint library is GPL licensed, but is only used for pre-build
> > >
> > > Python
> > >
> > > style checks and is neither bundled with, nor relied upon by, the Her=
on
> > > source or binary release artifacts.
> > >
> > > =3D=3D Cryptography =3D=3D
> > >
> > > Heron does not use any cryptography libraries.
> > >
> > > =3D Required Resources =3D
> > >
> > > =3D=3D Mailing lists =3D=3D
> > >
> > > * private@heron.incubator.apache.org <javascript:;> (with moderated
> > >
> > > subscriptions)
> > >
> > > * dev@heron.incubator.apache.org <javascript:;>
> > > * commits@heron.incubator.apache.org <javascript:;>
> > > * user@heron.incubator.apache.org <javascript:;>
> > >
> > >
> > > =3D=3D Subversion Directory =3D=3D
> > >
> > > Git is the preferred source control system: git://git.apache.org/hero=
n
> > >
> > > =3D=3D Issue Tracking =3D=3D
> > >
> > > JIRA: Heron (HERON)
> > >
> > > =3D=3D Initial Committers =3D=3D
> > >
> > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > * Bill Graham (billgraham at apache dot org)*
> > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > * Chris Kellogg (cckellogg at gmail dot com)
> > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > * Karthik Ramasamy (karthik at gmail dot com)
> > > * Maosong Fu (maosongfu at gmail dot com)
> > > * Neng Lu(freeneng at gmail dot com)
> > > * Runhang Li (obj dot runhang at gmail dot com)
> > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > * Supun Kamburugamuve (supun at apache dot org)*
> > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > >
> > > =3D=3D Affiliations =3D=3D
> > >
> > > * Andrew Jorgensen (Google)
> > > * Ashvin Agrawal (Microsoft)
> > > * Avrilia Floratou (Microsoft)
> > > * Bill Graham (Twitter)
> > > * Brian Hatfield (Google)
> > > * Chris Kellogg (Twitter)
> > > * Huijun Wu (Twitter)
> > > * Karthik Ramasamy (Streamlio)
> > > * Maosong Fu (Twitter)
> > > * Neng Lu (Twitter)
> > > * Runhang Li (Twitter)
> > > * Sanjeev Kulkarni (Streamlio)
> > > * Supun Kamburugamuve (Indiana University)
> > > * Thomas Sun (Twitter)
> > > * Yaliang Wang (Twitter)
> > >
> > > =3D Sponsors =3D
> > >
> > > =3D=3D Champion =3D=3D
> > >
> > > * Julien Le Dem (julien at apache dot org)
> > >
> > > =3D=3D Nominated Mentors =3D=3D
> > >
> > > * Jake Farrell (jfarrell at apache dot org)
> > > * Jacques Nadeau (jacques at apache dot org)
> > > * Julien Le Dem (julien at apache dot org)
> > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > >
> > > =3D=3D Sponsoring Entity =3D=3D
> > >
> > > The Apache Incubator
> > >
> > > =3D=3D Footnotes =3D=3D
> > >
> > > * 1 - Papers detailing Heron are available at
> > > http://dl.acm.org/citation.cfm?id=3D2742788 and
> > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > * 2 - http://home.apache.org/phonebook.html?uid=3Dbillgraham
> > > * 3 - http://home.apache.org/phonebook.html?uid=3Dashvin
> > > * 4 - http://home.apache.org/phonebook.html?uid=3Dsupun
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >
> > > <javascript:;>
> > >
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > > <javascript:;>
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sorry this was sent from mobile. Will do less grammar and spell check
> > than
> > > usual.
> > >
> > >
> > >
> >
>

--94eb2c1cf1ba65391e0552a0f248--