streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ate Douma <...@douma.nu>
Subject Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
Date Thu, 03 Mar 2016 15:48:14 GMT
+1 to everything below :)

Good stuff Steve!

On 2016-03-01 23:50, Steve Blackmon wrote:
> Following up:
>
> I took a pass at cleaning up the release markdown files and they now look
> mostly the same as they did under apache cms.
>
> I also confirmed that site-deploy can write to a sub-directory of
> streams.incubator.apache.org using scm:svn capability.
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> Recall that streams-project and streams-examples each publish schema
> artifacts to their maven sites, so maintaining prior versions is important.
> streams-master doesn't currently, but since it's a parent of those other
> two and prior experience has led to believe we're better off using the same
> scheme and setting the streams-master <url/> in the pom to be a sibling of
> streams-project and streams-example rather that publishing it to a
> different URL entirely.  This is why the version an artifactId are being
> included in the above url.
>
> Brings up two issues -
> resolving the latest release and/or snapshot version at any given time
> ensuring a visitor to streams.incubator.apache.org lands on the
> streams-master site for that version
>
> While symlinks are a viable way to resolving the latest release and/or
> snapshot version while keeping previous release versions around
>
> i.e. forwarding
> http://streams.incubator.apache.org/site/latest/streams-master/index.html
> to
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> they won't help us forward http://streams.incubator.apache.org to a
> subdirectory of itself.
>
> Good news is apache cms supports .htaccess and the following snippet can.
>
> RewriteEngine *on*
>
> Redirect /
> http://streams.incubator.apache.org/site/latest/streams-master/index.html
> Feel free to confirm by visiting streams.staging.incubator.org (you'll be
> redirected to the output of the maven site-deploy)
>
> *I ask that everyone on the list review and send me any concerns or
> feedback they want incorporated before this goes live on
> streams.incubator.apache.org <http://streams.incubator.apache.org>.*
>
> Procedurally, the way I expect that to happen is:
> - I'll submit a PR to streams-master/master
> - That PR will merge
> - Update the streams-master jenkins job to run site-deploy, and do the
> needful to get it the necessary credentials
> - Going forward all builds of streams-master/master will publish to
> site/${version}/streams-master
> - Add a few more instructions to jenkins to maintain the symlinks and
> .htaccess
> - If there are any manual steps required to oversee, document that on the
> site itself.
>
> Then, we'll move on to hooking streams-project and streams-examples up to
> this new site (as children with an integrated breadcrumb) and to jenkins CD.
>
> Steve Blackmon
> sblackmon@apache.org
>
> On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <sblackmon@apache.org>
> wrote:
>
>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>
>>
>> I agree - a diagram depicting streams mediating between sources and
>> destination on the landing page is appropriate.  I'll work on this soon.
>>
>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>
>>
>> I've pushed a new version (to the source and site links from my prior
>> message) that brings most of the existing content along, using the
>> reporting-info plugin to generate the HTML where possible.
>>
>> On a few of the pages - SCM, CI, and Dependency Info - the output of the
>> maven plugin simply isn't adequate for a multi-repository project such as
>> this, so those are now managed within src/site/markdown
>>
>> Release Setup and Release Process need some cleanup but all of the content
>> has been migrated.
>>
>> Other report-info pages - such as Dependencies, Dependency Management,
>> Dependency Convergence, JavaDocs, Test JavaDocs,  - don't make sense at the
>> streams-master level but do exist and are useful within streams-project,
>> streams-examples, and many of their respective sub-modules.
>>
>> In theory those additional report items will show up when navigating into
>> the site hierarchy of the sub-projects (links to each sub-project are
>> present along the header), which will inherit the structure, look-and-feel,
>> and breadcrumb of this site revision.  Naturally these changes must be
>> deployed to streams.incubator.apache.org and sub-project pom.xml and
>> site_en.xml will need some tweaking but I'm optimistic it will all fit
>> together nicely.
>>
>> Steve Blackmon
>> sblackmon@apache.org
>>
>> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <ate@douma.nu> wrote:
>>
>>> On 2016-02-25 23:50, Steve Blackmon wrote:
>>>
>>>> Hello streamers,
>>>>
>>>> I finally consolidated some materials I've been working on into a
>>>> nice-looking thing that is a viable candidate to replace the current
>>>> streams web page.  Please take a look and let me know what you think.
>>>> Sometime
>>>> soon I expect to submit a vote to take down the previous site and replace
>>>> it with something derived from this version.
>>>>
>>>> The proposed new site is temporarily being hosted at:
>>>>
>>>> http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>>
>>>> I placed all of the content into markdown files which are checked into a
>>>> branch of streams-master.  This approach allows us to author the site
>>>> using
>>>> markdown, package and deploy it with maven-site, and manage changes using
>>>> git.
>>>>
>>>
>>> This looks cool!
>>>
>>> I like the new overview and faq pages, definitely more concrete about
>>> Streams.
>>>
>>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>>
>>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>>
>>>
>>>> If you'd like to submit specific changes feel free to do so as
>>>> pull-requests to:
>>>> https://github.com/steveblackmon/incubator-streams-master.git
>>>>
>>>> Referring back to the original list of goals, some are addressed in whole
>>>> or in part with this revision, others not just yet.
>>>>
>>>> ADDRESSED IN WHOLE:
>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>> use
>>>> Streams
>>>> - there is no public javadoc or other technical documentation
>>>> linked/hosted
>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>
>>>> ADDRESSED IN PART:
>>>> - its missing concrete examples and recognizable use-cases for which
>>>> Streams might
>>>> be used, nor comparison against other solutions, and why Streams would
>>>> be a
>>>> good/better solution
>>>> - the Architecture page only has some basic wording but provides no
>>>> insight
>>>> whatsoever about the concrete implementation and certainly not its
>>>> architecture
>>>>
>>>> This is just a start.  Communication on the list has been dead lately,
>>>> but
>>>> I'm hoping to change that beginning with this effort to explain what is
>>>> unique and valuable about Streams, and why.  Hopefully this take on the
>>>> project will inspire others to get on-board and help out in the coming
>>>> months.
>>>>
>>>
>>> +1, good work!
>>>
>>>
>>>
>>>> Steve Blackmon
>>>> sblackmon@apache.org
>>>>
>>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <ate@douma.nu> wrote:
>>>>
>>>> Hi Steve,
>>>>>
>>>>> Cool. I don't have much extra input or ideas right now, but looking
>>>>> forward to the steps taken you proposed below.
>>>>> And about getting out a next release first soon, if it is not
>>>>> side-tracking these other steps much, sure +1
>>>>>
>>>>> Ate
>>>>>
>>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>>>>>
>>>>> All,
>>>>>>
>>>>>> Ate gave us a reality check during our board report in December and
>>>>>> enumerated some of the most prominent ways our documentation currently
>>>>>> falls short - of Apache community standards and of any reasonable
>>>>>> useful-ness test.  I've responded in-line below - please kick in
any
>>>>>> ideas
>>>>>> you have.
>>>>>>
>>>>>>
>>>>>> - the website isn't providing any practical guidance *why* or *how*
to
>>>>>> use
>>>>>> Streams
>>>>>>
>>>>>>
>>>>>> I agree.  This is the content that belongs on the streams website
>>>>>> landing
>>>>>> page (high-level) or linked from the body (the details).  I will
take a
>>>>>> first shot at this and circulate for feedback.
>>>>>>
>>>>>> - its missing concrete examples and recognizable use-cases for which
>>>>>> Streams might be used, nor comparison against other solutions, and
why
>>>>>> Streams would be a good/better solution
>>>>>>
>>>>>>
>>>>>> The examples documentation site [1] are meant to serve as both concrete
>>>>>> examples and recognizable use-cases, but there are a few problems
with
>>>>>> the
>>>>>> current setup.
>>>>>>
>>>>>>       1. The examples site lacks plain-language READMEs for the root
>>>>>> module
>>>>>>       and the first children of the root module (the runtime-specific
>>>>>>       aggregators).
>>>>>>       2. The examples site isn't linked from the landing page of
the
>>>>>> website.
>>>>>>       3. There is no permanent url for latest/current.
>>>>>>       4. There hasn't been a formal release of this repo.
>>>>>>       5. There are broken links within the markdown of some examples.
>>>>>>       6. The plain-language descriptions of the examples themselves
>>>>>> could be
>>>>>>
>>>>>>       improved.
>>>>>>
>>>>>> All of these can be addressed (easily I think) and I'm happy to take
>>>>>> them
>>>>>> on.
>>>>>>
>>>>>> Additionally, a page (or several) comparing streams to frameworks
such
>>>>>> as
>>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>>>>>> different/complementary would be valuable.  These questions come
up a
>>>>>> lot.
>>>>>>
>>>>>> - the Architecture page only has some basic wording but provides
no
>>>>>> insight
>>>>>> whatsoever about the concrete implementation and certainly not its
>>>>>> architecture
>>>>>>
>>>>>>
>>>>>> I think this page needs either an multi-page outline format heavily
>>>>>> linked
>>>>>> into READMEs and interfaces throughout the streams repos, or a system
>>>>>> diagram depicting a simple streams deployment alongside the vocabulary.
>>>>>> Maybe both as several pages.
>>>>>>
>>>>>> - there is no public javadoc or other technical documentation
>>>>>> linked/hosted
>>>>>>
>>>>>>
>>>>>> We do generate java-docs for releases and snapshots.  We should add
a
>>>>>> link
>>>>>> to the root page.  We've made progress but there are still many classes
>>>>>> lacking any javadoc header.  We should reiterate a commitment to
fixing
>>>>>> that and a schedule for deprecating and deleting any class that doesn't
>>>>>> still serve a distinct purpose.
>>>>>>
>>>>>> An index of the many resource files (json/xml schemas, conversion
>>>>>> rules,
>>>>>> shell scripts, etc...) published to the project and examples release
>>>>>> should
>>>>>> also placed on the main website.  Ideally comprehensive, but if not
at
>>>>>> least enough to show a developer that they exist and can be
>>>>>> hard-linked to
>>>>>> within their own project resources.
>>>>>>
>>>>>>
>>>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>>>
>>>>>>
>>>>>> This link (actually most of streams.incubator.apache.org) predates
the
>>>>>> regular publication of the streams-project maven site where READMEs
and
>>>>>> other resources are hosted.  Personally I'd prefer to see documentation
>>>>>> improving within the code-base rather than in a separate wiki in
the
>>>>>> near-term and suggest we just delete this link.
>>>>>>
>>>>>> - the mailing list is NOT used for any form of discussion/information
>>>>>> other
>>>>>> than JIRA and Git(hub) notifications
>>>>>> - especially the latter doesn't give a newbie any indication what
is
>>>>>> going
>>>>>> on, nor how to (start) interact
>>>>>>
>>>>>> This is unfortunate and must change for the community to grow.  I
know
>>>>>> I'm
>>>>>> guilty of having ad-hoc off-list discussions with my team that lead
to
>>>>>> new
>>>>>> stories and pull requests.  Mea culpa, I resolve to stop doing this
in
>>>>>> 2016.
>>>>>>
>>>>>> - there are no blog posts (that I am aware of) around using/trying
out
>>>>>> Streams either
>>>>>>
>>>>>>
>>>>>> This is a critical missing piece of the puzzle.  Static HTML via
apache
>>>>>> httpd is not the ideal platform for hosting a blog but some other
>>>>>> projects
>>>>>> have made it work.  I'd like us to replace 'Latest News' with a
>>>>>> navigable,
>>>>>> indexable set of blog posts going forward, where release summaries
and
>>>>>> new
>>>>>> examples can be published, and 'in-the-wild' mentions of the project
>>>>>> can
>>>>>> be
>>>>>> aggregated.
>>>>>>
>>>>>>
>>>>>> And I personally would prefer the dev@ list to be much more /
>>>>>> primarily
>>>>>> used for actual human interaction, not just these notifications from
>>>>>> JIRA
>>>>>> and Github.
>>>>>> Maybe change the configuration to delegate (some of) these to the
>>>>>> commit@
>>>>>> list instead?
>>>>>>
>>>>>>
>>>>>> I agree.  AFAICT though commit@streams.incubator.apache.org does
not
>>>>>> exist.  Per http://apache.org/dev/committers.html#mail we should
hold
>>>>>> a
>>>>>> vote to create it.  I'll open that vote today.
>>>>>>
>>>>>> What I suggest is to for a while stop (or largely postpone) working
on
>>>>>> code
>>>>>> but instead working on any/all of the above points first...
>>>>>>
>>>>>>
>>>>>> I agree that making the website helpful is the top project priority.
>>>>>>
>>>>>> Even working on a next release IMO is far less important than first
>>>>>> building up some explanation *why* a release of Stream is needed.
>>>>>>
>>>>>>
>>>>>> I agree that these concerns should be addressed before the next
>>>>>> release.
>>>>>> The main reason for releasing soon is that per [4] there are 36 issues
>>>>>> resolved since the 0.2 release, including quite a few bugs and
>>>>>> improvements
>>>>>> related to stability, deployment flexibility, or likely to impact
the
>>>>>> experience of a new developer.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>> http://streams.incubator.apache.org/site/0.2-incubating-SNAPSHOT/streams-examples/
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/apidocs/index.html
>>>>>> [3]
>>>>>>
>>>>>>
>>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/index.html
>>>>>> [4]
>>>>>>
>>>>>>
>>>>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


Mime
View raw message