streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Blackmon <st...@blackmon.org>
Subject Re: [DISCUSS] Continuing the Momentum
Date Thu, 17 Apr 2014 14:28:29 GMT
Standards-based stream processing - predictable, performant, at scale

?

On Thu, Apr 17, 2014 at 8:26 AM, Matt Franklin <m.ben.franklin@gmail.com> wrote:
> On Mon, Apr 14, 2014 at 5:22 PM, Renato MarroquĂ­n Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
>> Hi devs,
>>
>> Yeah the title was indeed compelling. You got me on that one lol
>> I think that you guys are right saying that for attracting new people maybe
>> we should try making the project's goal something more applicable in real
>> life than just being "a Lightweight server for ActivityStreams".
>> I liked the simple explanation I heard,maybe it was the pisco but please
>> correct me if I am wrong, "it's an abstraction layer for stream processing
>> engines". IMHO we have two things defined:
>>
>> MISION:
>> 1)  A flexible data processing framework that can run in multiple different
>> runtimes.  The goal being to abstract platform complexity and allow for
>> business logic reuse across real-time, enterprise, web and stand-alone
>> executions.
>>
>> This is what needs to be done.
>
>
>> VISION:
>> 2)  As a proving ground for the adoption of data format standards,
>> specifically ActivityStreams to start.  The community would work to drive
>> the adoption and evolution of such standards through real-world experience.
>>
>> This is where we would like to get at some time. But also to get more
>> community engaged, things have to simple. That is a big issue we still have
>> over in Gora, and we are trying to solve it through talks, better
>> tutorials, integration with other projects, and so forth.
>> Just my 2cents guys.
>>
>
> So what is the tag line that sums up both the mission and the vision?
>
>
>>
>>
>> Renato M.
>>
>>
>> 2014-04-14 16:31 GMT+02:00 Matt Franklin <m.ben.franklin@gmail.com>:
>>
>> > On Fri, Apr 11, 2014 at 5:01 PM, Steve Blackmon <sblackmon@apache.org
>> > >wrote:
>> >
>> > > On Thu, Apr 10, 2014 at 4:11 PM, Matt Franklin <
>> m.ben.franklin@gmail.com
>> > >
>> > > wrote:
>> > > > tl;dr version:
>> > > >
>> > > > We need to discuss things on the list more and work to define
>> streams,
>> > > > update our public presence to support this definition and encourage
>> > > > additional engagement.
>> > > >
>> > > +1, +1, +1
>> > >
>> > > > Long version:
>> > > >
>> > > > For those of you unaware, Steve Blackmon gave a nice talk on the work
>> > he
>> > > > has been committing to Streams at ApacheCon.  As part of that talk
>> and
>> > > > follow on discussions, it became clear that we as a community need
to
>> > do
>> > > > some serious work to define ourselves, what we are building and why
>> it
>> > is
>> > > > valuable to the industry.
>> > > >
>> > > If anyone who missed the presentation wants to see it, I'm happy to
>> > > host a google hangout to run through it.
>> > >
>> >
>> > Can you post it, or a link to it, on the website too?
>> >
>> >
>> > >
>> > > > Our website says we are a Lightweight server for ActivityStreams.
>> >  While
>> > > > this is true to some degree, I think recent contributions should
>> refine
>> > > > this.  The new code is really about supporting flexible processing,
>> > > > persistence and retrieval of data in multiple runtimes using strongly
>> > > > typed, normalized data formats like ActivityStreams.  Personally,
I
>> > think
>> > > > this slightly new direction is extremely compelling, and the reaction
>> > to
>> > > > Steve's talk seems to support that.  The question remains how does
>> the
>> > > > community as a whole see the project?  What value is everyone wanting
>> > to
>> > > > get out of this effort?
>> > > >
>> > > The session tag-line which attracted ~20 attendees was 'Simplifying
>> > > Real-Time data integration with Apache Streams.' From talking to
>> > > coders and data scientists I always hear frustration with how much
>> > > time they spend writing code and workflow to move bytes around and
>> > > keep track of their data assets. I'd wager any survey of prominent
>> > > open-source libraries and popular commercial APIs would have to
>> > > conclude that schema and interface standards are completely absent
>> > > or sparsely adopted at many layers.
>> > >
>> > > Standards in hardware, operating systems, networks, and relational
>> > > databases brought about flourishing ecosystems. I believe standards in
>> > > data interchange such as ActivityStreams can do the same for the
>> > > social web, but not everyone will embrace standards for the sake of
>> > > standards. If we can offer integration points to the data sources and
>> > > repositories businesses want to work with, and demonstrate that
>> > > Streams can handle 'fire-hose' scale data volumes with arbitrarily
>> > > many intermediate hand-offs and processing steps on messages in
>> > > flight, I think we will see adoption from enterprises looking to
>> > > replace ESB-type systems that can't keep up with the volume of data
>> > > generated (both inside and outside their networks) that they want to
>> > > track.  Streams is pretty decent at ETL as well - a function that is
>> > > never going away, even as the underlying tools best suited to
>> > > performing it at scale constantly change.
>> > >
>> > > This future-state I'm attempting to describe will be a better one for
>> > > researchers, hobbyists, entrepreneurs, and consumers of web products
>> > > and services.  Configuration-driven, runtime-platform agnostic,
>> > > software for real-time data exchange:  where community-driven
>> > > standards such as Activity Streams can codify and evolve
>> > > best-practices via running code.  That is a vision that I think will
>> > > help us generate significant traction going forward.
>> > >
>> >
>> > Just to make sure I am understanding you correctly, you are proposing we
>> > update the mission of the project to the following:
>> >
>> > 1)  A flexible data processing framework that can run in multiple
>> different
>> > runtimes.  The goal being to abstract platform complexity and allow for
>> > business logic reuse across real-time, enterprise, web and stand-alone
>> > executions.
>> > 2)  As a proving ground for the adoption of data format standards,
>> > specifically ActivityStreams to start.  The community would work to drive
>> > the adoption and evolution of such standards through real-world
>> experience.
>> >
>> > This sounds great, though it is slightly different than the initially
>> > proposed functionality.  Personally, I have no objection to that, as what
>> > you describe encompasses the original goals and expands on them; but, it
>> > would be good for the rest of the community to weigh in.
>> >
>> >
>> > >
>> > > > The fact that there are not clear answers (and corresponding
>> documented
>> > > > statements on the website) to these questions already means we are
>> not
>> > > > doing a great job of following the Apache Way.  The Apache Way is
>> about
>> > > the
>> > > > community and meritocratic, community-based decision making.  The
ASF
>> > > > defines it as follows:
>> > > >
>> > > > While there is not an official list, these six principles have been
>> > cited
>> > > > as the core beliefs of philosophy behind the foundation, which is
>> > > normally
>> > > > referred to as "The Apache Way":
>> > > >
>> > > > collaborative software development
>> > > >
>> > > > commercial-friendly standard license
>> > > >
>> > > > consistently high quality software
>> > > >
>> > > > respectful, honest, technical-based interaction
>> > > >
>> > > > faithful implementation of standards
>> > > >
>> > > > security as a mandatory feature
>> > > >
>> > > > All of the ASF projects share these principles.
>> > > >
>> > > > Let's make sure we propose changes to the list, create tickets that
>> > > support
>> > > > wider efforts and leverage principles like lazy consensus to keep
>> > moving
>> > > > forward in a way that supports the community.
>> > > +1, +1, +1
>> > >
>> > > On Thu, Apr 10, 2014 at 4:11 PM, Matt Franklin <
>> m.ben.franklin@gmail.com
>> > >
>> > > wrote:
>> > > > tl;dr version:
>> > > >
>> > > > We need to discuss things on the list more and work to define
>> streams,
>> > > > update our public presence to support this definition and encourage
>> > > > additional engagement.
>> > > >
>> > > > Long version:
>> > > >
>> > > > For those of you unaware, Steve Blackmon gave a nice talk on the work
>> > he
>> > > > has been committing to Streams at ApacheCon.  As part of that talk
>> and
>> > > > follow on discussions, it became clear that we as a community need
to
>> > do
>> > > > some serious work to define ourselves, what we are building and why
>> it
>> > is
>> > > > valuable to the industry.
>> > > >
>> > > > Our website says we are a Lightweight server for ActivityStreams.
>> >  While
>> > > > this is true to some degree, I think recent contributions should
>> refine
>> > > > this.  The new code is really about supporting flexible processing,
>> > > > persistence and retrieval of data in multiple runtimes using strongly
>> > > > typed, normalized data formats like ActivityStreams.  Personally,
I
>> > think
>> > > > this slightly new direction is extremely compelling, and the reaction
>> > to
>> > > > Steve's talk seems to support that.  The question remains how does
>> the
>> > > > community as a whole see the project?  What value is everyone wanting
>> > to
>> > > > get out of this effort?
>> > > >
>> > > > The fact that there are not clear answers (and corresponding
>> documented
>> > > > statements on the website) to these questions already means we are
>> not
>> > > > doing a great job of following the Apache Way.  The Apache Way is
>> about
>> > > the
>> > > > community and meritocratic, community-based decision making.  The
ASF
>> > > > defines it as follows:
>> > > >
>> > > > While there is not an official list, these six principles have been
>> > cited
>> > > > as the core beliefs of philosophy behind the foundation, which is
>> > > normally
>> > > > referred to as "The Apache Way":
>> > > >
>> > > > collaborative software development
>> > > >
>> > > > commercial-friendly standard license
>> > > >
>> > > > consistently high quality software
>> > > >
>> > > > respectful, honest, technical-based interaction
>> > > >
>> > > > faithful implementation of standards
>> > > >
>> > > > security as a mandatory feature
>> > > >
>> > > > All of the ASF projects share these principles.
>> > > >
>> > > > Let's make sure we propose changes to the list, create tickets that
>> > > support
>> > > > wider efforts and leverage principles like lazy consensus to keep
>> > moving
>> > > > forward in a way that supports the community.
>> > >
>> > >
>> > >
>> > > --
>> > > Steve Blackmon
>> > > sblackmon@apache.org
>> > >
>> >
>>

Mime
View raw message