Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <57211eb102e20e00002e99b1@polymail.io>
References: <56D85CBE.8030806@douma.nu>
	<57211eb102e20e00002e99b1@polymail.io>
Date: Wed, 27 Apr 2016 23:38:11 +0200
Message-ID: 
 <CAHYFERCG2De9XMtW1FKkjdbP6nRT8RNVDYYQWWjSDjGk8WLR0Q@mail.gmail.com>
Subject: Re: Streams Web Site, Maven Sites,
 misc Documentation - proposed next steps
From: =?UTF-8?Q?Renato_Marroqu=C3=ADn_Mogrovejo?=
 <renatoj.marroquin@gmail.com>
To: dev@streams.incubator.apache.org
Content-Type: multipart/alternative; boundary=047d7b624b6cedee0205317e39ba

--047d7b624b6cedee0205317e39ba
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Cool stuff Steve! Thanks!


Renato M.

2016-04-27 22:28 GMT+02:00 Steve Blackmon <sblackmon@apache.org>:

> All,
>
> I think we are pretty close to pulling all of the work on STREAMS-401 (ne=
w
> web page) together into a pretty slick package.  Thanks for your patience
> and feedback.
>
> Some new additions include diagrams on the architecture page and within
> many modules depicting streams components and data flow among them and wi=
th
> third-party systems, a 5-step tutorial to run a useful stream, consistent
> look-and-feel across all three repos, integrated breadcrumbs, and
> rudimentary github and twitter integrations.
>
> Feel free to browse these links to see where we=E2=80=99re at.
>
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-=
master/index.html
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-=
project/index.html
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-=
examples/index.html
>
> All of the menu link now resolve, or will resolve as soon as everything i=
s
> promoted.  If you get any Not Available errors that you CAN=E2=80=99T res=
olve by
> substituting =E2=80=98latest=E2=80=99 for =E2=80=980.3-incubating-SNAPSHO=
T=E2=80=99 in the url, please let
> me know.
>
> Additionally if you identify any major issues or have any last suggestion=
s
> please let me know before I kickoff the process to merge the STREAMS-401
> branches and promote the site later this week.
>
> Steve Blackmon
> sblackmon@apache.org
>
> On Thu, Mar 03, 2016 at 9:48 AM Ate Douma <Ate Douma
> <Ate+Douma+%3Cate@douma.nu%3E>> wrote:
>
>> +1 to everything below :)
>>
>> Good stuff Steve!
>>
>> On 2016-03-01 23:50, Steve Blackmon wrote:
>> > Following up:
>> >
>> > I took a pass at cleaning up the release markdown files and they now
>> look
>> > mostly the same as they did under apache cms.
>> >
>> > I also confirmed that site-deploy can write to a sub-directory of
>> > streams.incubator.apache.org using scm:svn capability.
>> >
>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/strea=
ms-master/index.html
>>
>> >
>> > Recall that streams-project and streams-examples each publish schema
>> > artifacts to their maven sites, so maintaining prior versions is
>> important.
>> > streams-master doesn't currently, but since it's a parent of those
>> other
>> > two and prior experience has led to believe we're better off using the
>> same
>> > scheme and setting the streams-master in the pom to be a sibling of
>> > streams-project and streams-example rather that publishing it to a
>> > different URL entirely. This is why the version an artifactId are bein=
g
>> > included in the above url.
>> >
>> > Brings up two issues -
>> > resolving the latest release and/or snapshot version at any given time
>> > ensuring a visitor to streams.incubator.apache.org lands on the
>> > streams-master site for that version
>> >
>> > While symlinks are a viable way to resolving the latest release and/or
>> > snapshot version while keeping previous release versions around
>> >
>> > i.e. forwarding
>> > http://streams.incubator.apache.org/site/latest/streams-master/index.h=
tml
>>
>> > to
>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/strea=
ms-master/index.html
>>
>> >
>> > they won't help us forward http://streams.incubator.apache.org to a
>> > subdirectory of itself.
>> >
>> > Good news is apache cms supports .htaccess and the following snippet
>> can.
>> >
>> > RewriteEngine *on*
>> >
>> > Redirect /
>> > http://streams.incubator.apache.org/site/latest/streams-master/index.h=
tml
>>
>> > Feel free to confirm by visiting streams.staging.incubator.org (you'll
>> be
>> > redirected to the output of the maven site-deploy)
>> >
>> > *I ask that everyone on the list review and send me any concerns or
>> > feedback they want incorporated before this goes live on
>> > streams.incubator.apache.org <http://streams.incubator.apache.org>.*
>> >
>> > Procedurally, the way I expect that to happen is:
>> > - I'll submit a PR to streams-master/master
>> > - That PR will merge
>> > - Update the streams-master jenkins job to run site-deploy, and do the
>> > needful to get it the necessary credentials
>> > - Going forward all builds of streams-master/master will publish to
>> > site/${version}/streams-master
>> > - Add a few more instructions to jenkins to maintain the symlinks and
>> > .htaccess
>> > - If there are any manual steps required to oversee, document that on
>> the
>> > site itself.
>> >
>> > Then, we'll move on to hooking streams-project and streams-examples up
>> to
>> > this new site (as children with an integrated breadcrumb) and to
>> jenkins CD.
>> >
>> > Steve Blackmon
>> > sblackmon@apache.org
>> >
>> > On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <sblackmon@apache.org>
>> > wrote:
>> >
>> >> One thing which I think can 'spice up' the overview page a bit more
>> would
>> >>> be a
>> >>> diagram providing some very high-level interaction between data
>> sources
>> >>> and
>> >>> destinations and the streams role between them.
>> >>
>> >>
>> >> I agree - a diagram depicting streams mediating between sources and
>> >> destination on the landing page is appropriate. I'll work on this
>> soon.
>> >>
>> >> And before replacing the current site several existing pages need to
>> be
>> >>> moved to
>> >>> the new one. But once those are done I think this is great
>> improvement.
>> >>
>> >>
>> >> I've pushed a new version (to the source and site links from my prior
>> >> message) that brings most of the existing content along, using the
>> >> reporting-info plugin to generate the HTML where possible.
>> >>
>> >> On a few of the pages - SCM, CI, and Dependency Info - the output of
>> the
>> >> maven plugin simply isn't adequate for a multi-repository project suc=
h
>> as
>> >> this, so those are now managed within src/site/markdown
>> >>
>> >> Release Setup and Release Process need some cleanup but all of the
>> content
>> >> has been migrated.
>> >>
>> >> Other report-info pages - such as Dependencies, Dependency Management=
,
>> >> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense a=
t
>> the
>> >> streams-master level but do exist and are useful within
>> streams-project,
>> >> streams-examples, and many of their respective sub-modules.
>> >>
>> >> In theory those additional report items will show up when navigating
>> into
>> >> the site hierarchy of the sub-projects (links to each sub-project are
>> >> present along the header), which will inherit the structure,
>> look-and-feel,
>> >> and breadcrumb of this site revision. Naturally these changes must be
>> >> deployed to streams.incubator.apache.org and sub-project pom.xml and
>> >> site_en.xml will need some tweaking but I'm optimistic it will all fi=
t
>> >> together nicely.
>> >>
>> >> Steve Blackmon
>> >> sblackmon@apache.org
>> >>
>> >> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <ate@douma.nu> wrote:
>> >>
>> >>> On 2016-02-25 23:50, Steve Blackmon wrote:
>> >>>
>> >>>> Hello streamers,
>> >>>>
>> >>>> I finally consolidated some materials I've been working on into a
>> >>>> nice-looking thing that is a viable candidate to replace the curren=
t
>> >>>> streams web page. Please take a look and let me know what you think=
.
>> >>>> Sometime
>> >>>> soon I expect to submit a vote to take down the previous site and
>> replace
>> >>>> it with something derived from this version.
>> >>>>
>> >>>> The proposed new site is temporarily being hosted at:
>> >>>>
>> >>>>
>> http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-mast=
er/index.html
>> >>>>
>> >>>> I placed all of the content into markdown files which are checked
>> into a
>> >>>> branch of streams-master. This approach allows us to author the sit=
e
>> >>>> using
>> >>>> markdown, package and deploy it with maven-site, and manage changes
>> using
>> >>>> git.
>> >>>>
>> >>>
>> >>> This looks cool!
>> >>>
>> >>> I like the new overview and faq pages, definitely more concrete abou=
t
>> >>> Streams.
>> >>>
>> >>> One thing which I think can 'spice up' the overview page a bit more
>> would
>> >>> be a
>> >>> diagram providing some very high-level interaction between data
>> sources
>> >>> and
>> >>> destinations and the streams role between them.
>> >>>
>> >>> And before replacing the current site several existing pages need to
>> be
>> >>> moved to
>> >>> the new one. But once those are done I think this is great
>> improvement.
>> >>>
>> >>>
>> >>>> If you'd like to submit specific changes feel free to do so as
>> >>>> pull-requests to:
>> >>>> https://github.com/steveblackmon/incubator-streams-master.git
>> >>>>
>> >>>> Referring back to the original list of goals, some are addressed in
>> whole
>> >>>> or in part with this revision, others not just yet.
>> >>>>
>> >>>> ADDRESSED IN WHOLE:
>> >>>> - the website isn't providing any practical guidance *why* or *how*
>> to
>> >>>> use
>> >>>> Streams
>> >>>> - there is no public javadoc or other technical documentation
>> >>>> linked/hosted
>> >>>> - there is a 'wiki (coming soon)' link which never has been
>> activated
>> >>>>
>> >>>> ADDRESSED IN PART:
>> >>>> - its missing concrete examples and recognizable use-cases for whic=
h
>> >>>> Streams might
>> >>>> be used, nor comparison against other solutions, and why Streams
>> would
>> >>>> be a
>> >>>> good/better solution
>> >>>> - the Architecture page only has some basic wording but provides no
>> >>>> insight
>> >>>> whatsoever about the concrete implementation and certainly not its
>> >>>> architecture
>> >>>>
>> >>>> This is just a start. Communication on the list has been dead
>> lately,
>> >>>> but
>> >>>> I'm hoping to change that beginning with this effort to explain wha=
t
>> is
>> >>>> unique and valuable about Streams, and why. Hopefully this take on
>> the
>> >>>> project will inspire others to get on-board and help out in the
>> coming
>> >>>> months.
>> >>>>
>> >>>
>> >>> +1, good work!
>> >>>
>> >>>
>> >>>
>> >>>> Steve Blackmon
>> >>>> sblackmon@apache.org
>> >>>>
>> >>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <ate@douma.nu> wrote:
>> >>>>
>> >>>> Hi Steve,
>> >>>>>
>> >>>>> Cool. I don't have much extra input or ideas right now, but lookin=
g
>> >>>>> forward to the steps taken you proposed below.
>> >>>>> And about getting out a next release first soon, if it is not
>> >>>>> side-tracking these other steps much, sure +1
>> >>>>>
>> >>>>> Ate
>> >>>>>
>> >>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>> >>>>>
>> >>>>> All,
>> >>>>>>
>> >>>>>> Ate gave us a reality check during our board report in December
>> and
>> >>>>>> enumerated some of the most prominent ways our documentation
>> currently
>> >>>>>> falls short - of Apache community standards and of any reasonable
>> >>>>>> useful-ness test. I've responded in-line below - please kick in
>> any
>> >>>>>> ideas
>> >>>>>> you have.
>> >>>>>>
>> >>>>>>
>> >>>>>> - the website isn't providing any practical guidance *why* or
>> *how* to
>> >>>>>> use
>> >>>>>> Streams
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree. This is the content that belongs on the streams website
>> >>>>>> landing
>> >>>>>> page (high-level) or linked from the body (the details). I will
>> take a
>> >>>>>> first shot at this and circulate for feedback.
>> >>>>>>
>> >>>>>> - its missing concrete examples and recognizable use-cases for
>> which
>> >>>>>> Streams might be used, nor comparison against other solutions, an=
d
>> why
>> >>>>>> Streams would be a good/better solution
>> >>>>>>
>> >>>>>>
>> >>>>>> The examples documentation site [1] are meant to serve as both
>> concrete
>> >>>>>> examples and recognizable use-cases, but there are a few problems
>> with
>> >>>>>> the
>> >>>>>> current setup.
>> >>>>>>
>> >>>>>> 1. The examples site lacks plain-language READMEs for the root
>> >>>>>> module
>> >>>>>> and the first children of the root module (the runtime-specific
>> >>>>>> aggregators).
>> >>>>>> 2. The examples site isn't linked from the landing page of the
>> >>>>>> website.
>> >>>>>> 3. There is no permanent url for latest/current.
>> >>>>>> 4. There hasn't been a formal release of this repo.
>> >>>>>> 5. There are broken links within the markdown of some examples.
>> >>>>>> 6. The plain-language descriptions of the examples themselves
>> >>>>>> could be
>> >>>>>>
>> >>>>>> improved.
>> >>>>>>
>> >>>>>> All of these can be addressed (easily I think) and I'm happy to
>> take
>> >>>>>> them
>> >>>>>> on.
>> >>>>>>
>> >>>>>> Additionally, a page (or several) comparing streams to frameworks
>> such
>> >>>>>> as
>> >>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>> >>>>>> different/complementary would be valuable. These questions come u=
p
>> a
>> >>>>>> lot.
>> >>>>>>
>> >>>>>> - the Architecture page only has some basic wording but provides
>> no
>> >>>>>> insight
>> >>>>>> whatsoever about the concrete implementation and certainly not it=
s
>> >>>>>> architecture
>> >>>>>>
>> >>>>>>
>> >>>>>> I think this page needs either an multi-page outline format
>> heavily
>> >>>>>> linked
>> >>>>>> into READMEs and interfaces throughout the streams repos, or a
>> system
>> >>>>>> diagram depicting a simple streams deployment alongside the
>> vocabulary.
>> >>>>>> Maybe both as several pages.
>> >>>>>>
>> >>>>>> - there is no public javadoc or other technical documentation
>> >>>>>> linked/hosted
>> >>>>>>
>> >>>>>>
>> >>>>>> We do generate java-docs for releases and snapshots. We should ad=
d
>> a
>> >>>>>> link
>> >>>>>> to the root page. We've made progress but there are still many
>> classes
>> >>>>>> lacking any javadoc header. We should reiterate a commitment to
>> fixing
>> >>>>>> that and a schedule for deprecating and deleting any class that
>> doesn't
>> >>>>>> still serve a distinct purpose.
>> >>>>>>
>> >>>>>> An index of the many resource files (json/xml schemas, conversion
>> >>>>>> rules,
>> >>>>>> shell scripts, etc...) published to the project and examples
>> release
>> >>>>>> should
>> >>>>>> also placed on the main website. Ideally comprehensive, but if no=
t
>> at
>> >>>>>> least enough to show a developer that they exist and can be
>> >>>>>> hard-linked to
>> >>>>>> within their own project resources.
>> >>>>>>
>> >>>>>>
>> >>>>>> - there is a 'wiki (coming soon)' link which never has been
>> activated
>> >>>>>>
>> >>>>>>
>> >>>>>> This link (actually most of streams.incubator.apache.org)
>> predates the
>> >>>>>> regular publication of the streams-project maven site where
>> READMEs and
>> >>>>>> other resources are hosted. Personally I'd prefer to see
>> documentation
>> >>>>>> improving within the code-base rather than in a separate wiki in
>> the
>> >>>>>> near-term and suggest we just delete this link.
>> >>>>>>
>> >>>>>> - the mailing list is NOT used for any form of
>> discussion/information
>> >>>>>> other
>> >>>>>> than JIRA and Git(hub) notifications
>> >>>>>> - especially the latter doesn't give a newbie any indication what
>> is
>> >>>>>> going
>> >>>>>> on, nor how to (start) interact
>> >>>>>>
>> >>>>>> This is unfortunate and must change for the community to grow. I
>> know
>> >>>>>> I'm
>> >>>>>> guilty of having ad-hoc off-list discussions with my team that
>> lead to
>> >>>>>> new
>> >>>>>> stories and pull requests. Mea culpa, I resolve to stop doing thi=
s
>> in
>> >>>>>> 2016.
>> >>>>>>
>> >>>>>> - there are no blog posts (that I am aware of) around using/tryin=
g
>> out
>> >>>>>> Streams either
>> >>>>>>
>> >>>>>>
>> >>>>>> This is a critical missing piece of the puzzle. Static HTML via
>> apache
>> >>>>>> httpd is not the ideal platform for hosting a blog but some other
>> >>>>>> projects
>> >>>>>> have made it work. I'd like us to replace 'Latest News' with a
>> >>>>>> navigable,
>> >>>>>> indexable set of blog posts going forward, where release summarie=
s
>> and
>> >>>>>> new
>> >>>>>> examples can be published, and 'in-the-wild' mentions of the
>> project
>> >>>>>> can
>> >>>>>> be
>> >>>>>> aggregated.
>> >>>>>>
>> >>>>>>
>> >>>>>> And I personally would prefer the dev@ list to be much more /
>> >>>>>> primarily
>> >>>>>> used for actual human interaction, not just these notifications
>> from
>> >>>>>> JIRA
>> >>>>>> and Github.
>> >>>>>> Maybe change the configuration to delegate (some of) these to the
>> >>>>>> commit@
>> >>>>>> list instead?
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree. AFAICT though commit@streams.incubator.apache.org does
>> not
>> >>>>>> exist. Per http://apache.org/dev/committers.html#mail we should
>> hold
>> >>>>>> a
>> >>>>>> vote to create it. I'll open that vote today.
>> >>>>>>
>> >>>>>> What I suggest is to for a while stop (or largely postpone)
>> working on
>> >>>>>> code
>> >>>>>> but instead working on any/all of the above points first...
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree that making the website helpful is the top project
>> priority.
>> >>>>>>
>> >>>>>> Even working on a next release IMO is far less important than
>> first
>> >>>>>> building up some explanation *why* a release of Stream is needed.
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree that these concerns should be addressed before the next
>> >>>>>> release.
>> >>>>>> The main reason for releasing soon is that per [4] there are 36
>> issues
>> >>>>>> resolved since the 0.2 release, including quite a few bugs and
>> >>>>>> improvements
>> >>>>>> related to stability, deployment flexibility, or likely to impact
>> the
>> >>>>>> experience of a new developer.
>> >>>>>>
>> >>>>>> [1]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating-SNAPSHOT/=
streams-examples/
>>
>> >>>>>> [2]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-p=
roject/apidocs/index.html
>>
>> >>>>>> [3]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-p=
roject/index.html
>>
>> >>>>>> [4]
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> https://issues.apache.org/jira/issues/?jql=3Dproject%20%3D%20STREAMS%20A=
ND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unrelea=
sedVersions()
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>

--047d7b624b6cedee0205317e39ba--