streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sblackmon <>
Subject Re: Update to continuous integration flow
Date Fri, 21 Oct 2016 03:34:27 GMT
Jenkins jobs streams-project-verify and streams-examples-verify are passing as well now, including
all of the new live data collection, docker database, and flink example integration tests.

streams-project-verify is a bit particular only because the facebook credentials it’s been
given have a very quick expiration.  So until we source more stable credentials to use for
testing, a new token will need to be provider prior to any execution where we’re looking
for 100% integration test pass rate - probably just builds where something has changed in
that module or built from a release candidate.

streams-project-site and streams-examples-site have been configured to run on SCM updates,
but at the moment they are not setup to auto-deploy to the website - that’s still a documented
manual process.  You can however click to view the staging site in jenkins.

On October 18, 2016 at 10:58:07 PM, sblackmon ( wrote:


Jenkins jobs streams-master streams-project and streams-examples are blue with no test failures.

Jenkins jobs streams-project-verify and streams-examples-verify have been created and configured.

I’ve observed them successfully launching and reading/writing to docker containers, as well
as collecting data from twitter using credentials supplied by the job.

After merging the open PRs and a bit more tinkering in Jenkins, I expect those jobs to be
blue as well with all ITs passing project-wide.


On October 17, 2016 at 3:42:04 PM, sblackmon ( wrote:


I’m working on the streams jenkins setup to increase our CI coverage and support more types
of testing.

What’s changed code wise to precipitate changes?

- streams-master and streams-examples are now release artifacts and need CI
- some providers now contain integration tests that collect data live, but require working
api credentials to do so
- some persisters (and all examples) now contain integration tests that launch supporting
services with docker
- streams-project now contains modules that build maven plugins, and has testing that relies
on those plugins

What’s changing? 

- integration testing (mvn verify) is being isolated from standard build (mvn install)
- integration testing steps will be configured to launch docker containers and use test credentials
to run the live data tests
- snapshots and site snapshots will publish as separate steps, contingent on stability of
standard builds

Attached a diagram of the new build flow, which will be including on the continuous integration
page of the website.

Dashed line: checkout
Dotted line: jenkins upstream/downstream relationships
Solid line: deployment of artifacts

Hint: you can click each node in the SVG to jump to a relevant URL.

I’m working on setting this up and hope to have everything fully working before the 0.4
release.  In the meantime you may see build failures as jobs and test plans are being refactored
in STREAMS-429 branches.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message