Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAENpzYXkdUJroCBeG5pDjJD3n2672qYnaa23hYUFDRj-vLqoig@mail.gmail.com>
References: 
 <CAENpzYXkdUJroCBeG5pDjJD3n2672qYnaa23hYUFDRj-vLqoig@mail.gmail.com>
Date: Fri, 11 Apr 2014 17:01:39 -0600
Message-ID: 
 <CAJ7ACC-HrYDH4L5c9zkXb80kw9k4-pxxBT2DP_SE5OvBet_CFQ@mail.gmail.com>
Subject: Re: [DISCUSS] Continuing the Momentum
From: Steve Blackmon <sblackmon@apache.org>
To: dev@streams.incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Apr 10, 2014 at 4:11 PM, Matt Franklin <m.ben.franklin@gmail.com> wrote:
> tl;dr version:
>
> We need to discuss things on the list more and work to define streams,
> update our public presence to support this definition and encourage
> additional engagement.
>
+1, +1, +1

> Long version:
>
> For those of you unaware, Steve Blackmon gave a nice talk on the work he
> has been committing to Streams at ApacheCon.  As part of that talk and
> follow on discussions, it became clear that we as a community need to do
> some serious work to define ourselves, what we are building and why it is
> valuable to the industry.
>
If anyone who missed the presentation wants to see it, I'm happy to
host a google hangout to run through it.

> Our website says we are a Lightweight server for ActivityStreams.  While
> this is true to some degree, I think recent contributions should refine
> this.  The new code is really about supporting flexible processing,
> persistence and retrieval of data in multiple runtimes using strongly
> typed, normalized data formats like ActivityStreams.  Personally, I think
> this slightly new direction is extremely compelling, and the reaction to
> Steve's talk seems to support that.  The question remains how does the
> community as a whole see the project?  What value is everyone wanting to
> get out of this effort?
>
The session tag-line which attracted ~20 attendees was 'Simplifying
Real-Time data integration with Apache Streams.' From talking to
coders and data scientists I always hear frustration with how much
time they spend writing code and workflow to move bytes around and
keep track of their data assets. I'd wager any survey of prominent
open-source libraries and popular commercial APIs would have to
conclude that schema and interface standards are completely absent
or sparsely adopted at many layers.

Standards in hardware, operating systems, networks, and relational
databases brought about flourishing ecosystems. I believe standards in
data interchange such as ActivityStreams can do the same for the
social web, but not everyone will embrace standards for the sake of
standards. If we can offer integration points to the data sources and
repositories businesses want to work with, and demonstrate that
Streams can handle 'fire-hose' scale data volumes with arbitrarily
many intermediate hand-offs and processing steps on messages in
flight, I think we will see adoption from enterprises looking to
replace ESB-type systems that can't keep up with the volume of data
generated (both inside and outside their networks) that they want to
track.  Streams is pretty decent at ETL as well - a function that is
never going away, even as the underlying tools best suited to
performing it at scale constantly change.

This future-state I'm attempting to describe will be a better one for
researchers, hobbyists, entrepreneurs, and consumers of web products
and services.  Configuration-driven, runtime-platform agnostic,
software for real-time data exchange:  where community-driven
standards such as Activity Streams can codify and evolve
best-practices via running code.  That is a vision that I think will
help us generate significant traction going forward.

> The fact that there are not clear answers (and corresponding documented
> statements on the website) to these questions already means we are not
> doing a great job of following the Apache Way.  The Apache Way is about the
> community and meritocratic, community-based decision making.  The ASF
> defines it as follows:
>
> While there is not an official list, these six principles have been cited
> as the core beliefs of philosophy behind the foundation, which is normally
> referred to as "The Apache Way":
>
> collaborative software development
>
> commercial-friendly standard license
>
> consistently high quality software
>
> respectful, honest, technical-based interaction
>
> faithful implementation of standards
>
> security as a mandatory feature
>
> All of the ASF projects share these principles.
>
> Let's make sure we propose changes to the list, create tickets that support
> wider efforts and leverage principles like lazy consensus to keep moving
> forward in a way that supports the community.
+1, +1, +1

On Thu, Apr 10, 2014 at 4:11 PM, Matt Franklin <m.ben.franklin@gmail.com> wrote:
> tl;dr version:
>
> We need to discuss things on the list more and work to define streams,
> update our public presence to support this definition and encourage
> additional engagement.
>
> Long version:
>
> For those of you unaware, Steve Blackmon gave a nice talk on the work he
> has been committing to Streams at ApacheCon.  As part of that talk and
> follow on discussions, it became clear that we as a community need to do
> some serious work to define ourselves, what we are building and why it is
> valuable to the industry.
>
> Our website says we are a Lightweight server for ActivityStreams.  While
> this is true to some degree, I think recent contributions should refine
> this.  The new code is really about supporting flexible processing,
> persistence and retrieval of data in multiple runtimes using strongly
> typed, normalized data formats like ActivityStreams.  Personally, I think
> this slightly new direction is extremely compelling, and the reaction to
> Steve's talk seems to support that.  The question remains how does the
> community as a whole see the project?  What value is everyone wanting to
> get out of this effort?
>
> The fact that there are not clear answers (and corresponding documented
> statements on the website) to these questions already means we are not
> doing a great job of following the Apache Way.  The Apache Way is about the
> community and meritocratic, community-based decision making.  The ASF
> defines it as follows:
>
> While there is not an official list, these six principles have been cited
> as the core beliefs of philosophy behind the foundation, which is normally
> referred to as "The Apache Way":
>
> collaborative software development
>
> commercial-friendly standard license
>
> consistently high quality software
>
> respectful, honest, technical-based interaction
>
> faithful implementation of standards
>
> security as a mandatory feature
>
> All of the ASF projects share these principles.
>
> Let's make sure we propose changes to the list, create tickets that support
> wider efforts and leverage principles like lazy consensus to keep moving
> forward in a way that supports the community.


-- 
Steve Blackmon
sblackmon@apache.org