streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Blackmon <st...@blackmon.org>
Subject Re: provider refactoring/improvement exemplar
Date Tue, 11 Oct 2016 18:57:30 GMT
STREAMS-403 : run embedded, outside of a streams runtime
STREAMS-411 : run from console via main method


On Tue, Oct 11, 2016 at 1:43 PM, Matt Franklin <m.ben.franklin@gmail.com> wrote:
> On Thu, Oct 6, 2016 at 10:32 AM Steve Blackmon <steve@blackmon.org> wrote:
>
>> This issue order makes sense to me, based on simplicity and complexity
>>
>> of improvement:
>>
>>
>>
>> > > STREAMS-403(https://issues.apache.org/jira/browse/STREAMS-403)
>>
>> > > Ensure all providers function stand-alone
>>
>
> Just so I am following, what is the purpose of this?  Run standalone from a
> console or run standalone outside of a streams runtime.
>
>
>>
>> > > STREAMS-411(https://issues.apache.org/jira/browse/STREAMS-411)
>>
>> > > ability (and instructions on how) to run providers directly from
>> console
>>
>> > > STREAMS-422(https://issues.apache.org/jira/browse/STREAMS-422)
>>
>> > > SNS providers should emit appropriately typed documents by default
>>
>> > > STREAMS-415(https://issues.apache.org/jira/browse/STREAMS-415)
>>
>> > > Proof of concept integration test that pulls actual data from generator
>>
>>
>>
>> In terms of which providers to target in what order, I’d suggest:
>>
>>
>>
>> twitter
>>
>> instagram
>>
>> rss
>>
>> youtube
>>
>> facebook
>>
>> gplus
>>
>> sysomos
>>
>> moreover
>>
>>
>>
>> Further down the list: more uncertainly about what exactly to do, how
>>
>> to do it, how to live test it, higher likelihood of encountering
>>
>> technical debt.
>>
>>
>>
>> If anyone wants to take a shot at any piece of this, go for it.  No
>>
>> need for omni-bus PRs covering a full story or a full provider, every
>>
>> incremental improvement helps. I’ll happily code review any PR that’s
>>
>> opened and help get it finished.
>>
>>
>>
>> Steve
>>
>>
>>
>>
>>
>> On October 6, 2016 at 7:52:42 AM, Suneel Marthi
>>
>> (smarthi@apache.org(mailto:smarthi@apache.org)) wrote:
>>
>>
>>
>> > On Tue, Oct 4, 2016 at 9:58 PM, sblackmon wrote:
>>
>> >
>>
>> > >
>>
>> > > All,
>>
>> > >
>>
>> > > I’ve pushed a branch that depicts the sort of provider upgrades that
I
>>
>> > > envision making across all providers this quarter.
>>
>> > >
>>
>> > > https://github.com/apache/incubator-streams/compare/
>>
>> > > master...steveblackmon:STREAMS-415?expand=1
>>
>> > >
>>
>> > > It solves for all of the following clean-up and improvement issues,
>>
>> > > specifically for the TwitterTimelineProvider
>>
>> > >
>>
>> > > STREAMS-403(https://issues.apache.org/jira/browse/STREAMS-403)
>>
>> > > Ensure all providers function stand-alone
>>
>> > >
>>
>> > > STREAMS-422(https://issues.apache.org/jira/browse/STREAMS-422)
>>
>> > > SNS providers should emit appropriately typed documents by default
>>
>> > >
>>
>> > > STREAMS-411(https://issues.apache.org/jira/browse/STREAMS-411)
>>
>> > > ability (and instructions on how) to run providers directly from
>> console
>>
>> > >
>>
>> > > STREAMS-415(https://issues.apache.org/jira/browse/STREAMS-415)
>>
>> > > Proof of concept integration test that pulls actual data from generator
>>
>> > >
>>
>> > > Rather than tackle these stories one at a time laterally across all
>>
>> > > providers, we could consider divide-and-conquer by module or by class
>> to
>>
>> > > reduce the number of commits and committers hitting each provider.
>>
>> > >
>>
>> > > Curious what others want to try. Would you rather take on one provider
>> at
>>
>> > > a time and improve it’s quality across several dimensions, or improve
>> many
>>
>> > > at once in a single dimension?
>>
>> > >
>>
>> >
>>
>> > I think we should do one at a time given the available and committed
>>
>> > resources. Which one would u like to tackle fist from the list above?
>>
>> >
>>
>> > >
>>
>> > > Steve
>>
>> > >
>>
>> > > On October 4, 2016 at 11:56:18 AM, sblackmon (sblackmon@apache.org
>> (mailto:
>>
>> > > sblackmon@apache.org)) wrote:
>>
>> > >
>>
>> > > > With 0.3-incubating in the books, it’s time to take a look at what
is
>>
>> > > most important for the next release.
>>
>> > > >
>>
>> > > > Here are a few concrete things I propose we tackle in October. They
>> are
>>
>> > > tagged in JIRA with fixVersion=0.4
>>
>> > > >
>>
>> > > > https://issues.apache.org/jira/browse/STREAMS-418?filter=12338530
>>
>> > > >
>>
>> > > > I saw some new issues being created the last few days. Please keep
it
>>
>> > > up, no matter how small!
>>
>> > > >
>>
>> > > > Major themes:
>>
>> > > > * Reboot and Clean-up
>>
>> > > > * Provider and Configuration consistency
>>
>> > > > * Prep for the AS 2.0 switch
>>
>> > > > Also:
>>
>> > > > * POC persist integration tests with docker
>>
>> > > > * POC provider integration tests that perform collection
>>
>> > > > * Fix lingering and simple bugs / documentation
>>
>> > > > * Flink example code
>>
>> > > >
>>
>> > > > If we target early week of October 24 for code freeze and cut rc1
>> that
>>
>> > > should give us time to get the release passed (hopefully with unanimous
>>
>> > > support from three active IPMC members on PPMC) by our November report.
>>
>> > > >
>>
>> > > > Feedback welcome! And feel free to assign yourself or let us someone
>>
>> > > with permissions know if you want to have something assigned to you.
>>
>> > > >
>>
>> > > > Steve
>>
>> > > >
>>
>> > > > On September 28, 2016 at 3:00:01 PM, sblackmon (sblackmon@apache.org
>>
>> > > (mailto:sblackmon@apache.org)) wrote:
>>
>> > > >
>>
>> > > > > All,
>>
>> > > > >
>>
>> > > > > Joey brought this up over the weekend and I think a discussion
is
>>
>> > > overdue on the topic.
>>
>> > > > >
>>
>> > > > > Encouraging community growth and performing regular releases
are on
>>
>> > > our list of graduation criteria.
>>
>> > > > >
>>
>> > > > > A few easy behaviors we can adopt to take to make progress on
these
>>
>> > > goals:
>>
>> > > > > - planning release versions around one or two significant
>> improvements
>>
>> > > > > - setting target dates to kick off upcoming releases
>>
>> > > > > - prioritizing our backlog after each release
>>
>> > > > > - discussing project and community milestones openly on the list
>>
>> > > > > - organizing JIRA so that all contributors (especially new) can
>> decide
>>
>> > > where it’s most important to focus their efforts
>>
>> > > > >
>>
>> > > > > I think to get things moving again and demonstrate we are capable
>> of
>>
>> > > consistent progress, we should aim to perform a release once per month
>>
>> > > around the end of the month.
>>
>> > > > >
>>
>> > > > > As for what to focus on, I think it’s time to discuss adopting
>>
>> > > Activity Streams 2.0, figure out what form that transition would take,
>> and
>>
>> > > get started down that path. Working implementations demonstrate the
>>
>> > > suitability of the standard and drive it’s adoption, and the prospects
>> of
>>
>> > > this project are closely tied to those of the standard. Separate
>> DISCUSS
>>
>> > > coming on this topic.
>>
>> > > > >
>>
>> > > > > Also important for the ‘reboot’ theme, we should delete any
>> modules we
>>
>> > > aren’t going to maintain, and bring all modules we are going to
>> maintain up
>>
>> > > to acceptable standards - exactly what that means is an open question
>> but
>>
>> > > broadly they should have documentation, code comments, and tests at the
>>
>> > > level of a typical module in a typical TLP.
>>
>> > > > >
>>
>> > > > > Expanding the examples to demonstrate how to use streams providers
>> and
>>
>> > > processors within various execution engines and fixing any bugs that
>> have
>>
>> > > been reported is desirable as well. Adding at least one new example per
>>
>> > > release is a good target for now.
>>
>> > > > >
>>
>> > > > > I have created some future versions with target release dates
in
>> JIRA
>>
>> > > and invite all committers to associate existing or new issues with
>> those
>>
>> > > releases, or anyone who can’t modify JIRA to summarize their thoughts
>> and
>>
>> > > share with the list and I will incorporate those ideas into JIRA. This
>>
>> > > should be the default reference for anyone looking for a way to help -
>> look
>>
>> > > at issues associated with the next few releases and the top of the
>> backlog
>>
>> > > and pick something that appeals and is in line with your experience.
>>
>> > > > >
>>
>> > > > > Anything else that should be a top priority for the rest of the
>> year?
>>
>> > > Or other ideas on improving planning and coordination?
>>
>> > > > >
>>
>> > > > > Steve
>>
>> > > > >
>>
>> > > > > On September 24, 2016 at 1:01:02 PM, apache (sblackmon@apache.org
>>
>> > > (mailto:sblackmon@apache.org)) wrote:
>>
>> > > > > > - This has already come up, but maybe ActivityStreams 2.0
support
>>
>> > > would broaden the community and motivate more work. It's also a
>> concrete
>>
>> > > > >
>>
>> > > > >
>>
>> > > > > > goal to work toward so people would know where they can
start.
>>
>> > > > >
>>
>> > > > >
>>
>> > > > >
>>
>> > > > >
>>
>> > > > > > - Steve and I did a little work here a few months ago, but
the
>> JIRA
>>
>> > > could reflect the priorities better and I think keep the community
>> working
>>
>> > > in a common direction.
>>
>> > > > >
>>
>> > > > >
>>
>> > > > >
>>
>> > > > >
>>
>> > > > >
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > >
>>
>>

Mime
View raw message