Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
Date: Tue, 4 Oct 2016 20:58:18 -0500
From: sblackmon <sblackmon@apache.org>
To: dev@streams.incubator.apache.org
Message-ID: <etPan.57f45e3a.4d439b0e.49a7@apache.org>
In-Reply-To: <etPan.57f3df31.482b8cc4.49a7@apache.org>
References: <d1beb683-8f16-49a8-8a36-3d417a263f89@me.com>
 <etPan.57e6bf5d.281499b1.1154@apache.org>
 <etPan.57ec2140.7b27efd7.1154@apache.org>
 <etPan.57f3df31.482b8cc4.49a7@apache.org>
Subject: provider refactoring/improvement exemplar
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
archived-at: Wed, 05 Oct 2016 01:58:31 -0000

=C2=A0
All,

I=E2=80=99ve pushed a branch that depicts the sort of provider upgrades t=
hat I envision making across all providers this quarter. =20

https://github.com/apache/incubator-streams/compare/master...steveblackmo=
n:STREAMS-415=3Fexpand=3D1 =20

It solves for all of the following clean-up and improvement issues, speci=
fically for the TwitterTimelineProvider =20

STREAMS-403(https://issues.apache.org/jira/browse/STREAMS-403) =20
Ensure all providers function stand-alone

STREAMS-422(https://issues.apache.org/jira/browse/STREAMS-422)
SNS providers should emit appropriately typed documents by default

STREAMS-411(https://issues.apache.org/jira/browse/STREAMS-411) =20
ability (and instructions on how) to run providers directly from console

STREAMS-415(https://issues.apache.org/jira/browse/STREAMS-415)
Proof of concept integration test that pulls actual data from generator

Rather than tackle these stories one at a time laterally across all provi=
ders, we could consider divide-and-conquer by module or by class to reduc=
e the number of commits and committers hitting each provider. =20

Curious what others want to try. =C2=A0Would you rather take on one provi=
der at a time and improve it=E2=80=99s quality across several dimensions,=
 or improve many at once in a single dimension=3F

Steve

On October 4, 2016 at 11:56:18 AM, sblackmon (sblackmon=40apache.org(mail=
to:sblackmon=40apache.org)) wrote:

> With 0.3-incubating in the books, it=E2=80=99s time to take a look at w=
hat is most important for the next release.
> =20
> Here are a few concrete things I propose we tackle in October. They are=
 tagged in JIRA with fixVersion=3D0.4 =20
> =20
> https://issues.apache.org/jira/browse/STREAMS-418=3Ffilter=3D12338530 =20
> =20
> I saw some new issues being created the last few days. Please keep it u=
p, no matter how small=21 =20
> =20
> Major themes: =20
> * Reboot and Clean-up
> * Provider and Configuration consistency
> * Prep for the AS 2.0 switch
> Also:
> * POC persist integration tests with docker =20
> * POC provider integration tests that perform collection
> * =46ix lingering and simple bugs / documentation
> * =46link example code
> =20
> If we target early week of October 24 for code freeze and cut rc1 that =
should give us time to get the release passed (hopefully with unanimous s=
upport from three active IPMC members on PPMC) by our November report. =20
> =20
> =46eedback welcome=21 And feel free to assign yourself or let us someon=
e with permissions know if you want to have something assigned to you. =20
> =20
> Steve =20
> =20
> On September 28, 2016 at 3:00:01 PM, sblackmon (sblackmon=40apache.org(=
mailto:sblackmon=40apache.org)) wrote:
> =20
> > All,
> > =20
> > Joey brought this up over the weekend and I think a discussion is ove=
rdue on the topic. =20
> > =20
> > Encouraging community growth and performing regular releases are on o=
ur list of graduation criteria. =20
> > =20
> > A few easy behaviors we can adopt to take to make progress on these g=
oals: =20
> > - planning release versions around one or two significant improvement=
s
> > - setting target dates to kick off upcoming releases
> > - prioritizing our backlog after each release
> > - discussing project and community milestones openly on the list =20
> > - organizing JIRA so that all contributors (especially new) can decid=
e where it=E2=80=99s most important to focus their efforts
> > =20
> > I think to get things moving again and demonstrate we are capable of =
consistent progress, we should aim to perform a release once per month ar=
ound the end of the month. =20
> > =20
> > As for what to focus on, I think it=E2=80=99s time to discuss adoptin=
g Activity Streams 2.0, figure out what form that transition would take, =
and get started down that path. Working implementations demonstrate the s=
uitability of the standard and drive it=E2=80=99s adoption, and the prosp=
ects of this project are closely tied to those of the standard. Separate =
DISCUSS coming on this topic. =20
> > =20
> > Also important for the =E2=80=98reboot=E2=80=99 theme, we should dele=
te any modules we aren=E2=80=99t going to maintain, and bring all modules=
 we are going to maintain up to acceptable standards - exactly what that =
means is an open question but broadly they should have documentation, cod=
e comments, and tests at the level of a typical module in a typical TLP. =
=20
> > =20
> > Expanding the examples to demonstrate how to use streams providers an=
d processors within various execution engines and fixing any bugs that ha=
ve been reported is desirable as well. Adding at least one new example pe=
r release is a good target for now. =20
> > =20
> > I have created some future versions with target release dates in JIRA=
 and invite all committers to associate existing or new issues with those=
 releases, or anyone who can=E2=80=99t modify JIRA to summarize their tho=
ughts and share with the list and I will incorporate those ideas into JIR=
A. This should be the default reference for anyone looking for a way to h=
elp - look at issues associated with the next few releases and the top of=
 the backlog and pick something that appeals and is in line with your exp=
erience. =20
> > =20
> > Anything else that should be a top priority for the rest of the year=3F=
 Or other ideas on improving planning and coordination=3F =20
> > =20
> > Steve =20
> > =20
> > On September 24, 2016 at 1:01:02 PM, apache (sblackmon=40apache.org(m=
ailto:sblackmon=40apache.org)) wrote: =20
> > > - This has already come up, but maybe ActivityStreams 2.0 support w=
ould broaden the community and motivate more work. It's also a concrete =20
> > =20
> > =20
> > > goal to work toward so people would know where they can start.
> > =20
> > =20
> > =20
> > =20
> > > - Steve and I did a little work here a few months ago, but the JIRA=
 could reflect the priorities better and I think keep the community worki=
ng in a common direction.
> > =20
> > =20
> > =20
> > =20
> > =20
> =20
> =20
> =20
> =20