streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominik Riemer" <>
Subject CI Pipelines
Date Wed, 11 Dec 2019 21:55:26 GMT

I'm currently trying to figure out what could be the best CI strategy in the
ASF infrastructure.


For StreamPipes development, the following things are important for a good
development experience:

-        Users with different roles (e.g., UI dev, backend dev, pipeline
element dev) should be able to work independently on components without
needing to setup the whole development stack

-        We currently manage this in Docker, e.g., when you are developing
on the UI, all other components and services run in Docker, so that UI
developers don't need to care about how to setup a Maven project and vice

-        When developing pipeline elements (which are in the repo
incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes
artifacts need to be fetched from a Maven repository, so that pipeline
element development can happen based on the latest SNAPSHOT version


Currently, we use a build pipeline in Gitlab CI as follows (taking the
example of the core at incubator-streampipes):

-        Build Maven artifacts

-        Build UI

-        Build Docker Images and deploy these images to Docker Hub 

-        Deploy Maven artifacts to Nexus

This was quite a smooth and integrated process.


So after having talked to Chris and INFRA, it seems that only Jenkins is
allowed to publish to the Apache Maven Nexus, but Jenkins can only push to
the official Apache Docker Hub account. As our development is heavily
focused around Docker (we are building around 20 Docker Images for the
individual services), I'd personally prefer to use the 2nd way proposed in
the Docker Hub release policy (
, e.g., having an apache-streampipes organization account on Docker Hub) as
it gives users a much better overview on what images are currently
available. However, this won't work from Jenkins as it's not possible to
push to a non-ASF-namespace.


I played around a little bit with Jenkins pipelines and while Docker-based
builds are generally supported, I'm not yet 100% sure how easy it is to
share artifacts over stages.


So I guess we have two options, both approaches have pros and cons:

-        Use Jenkins for the complete pipeline and publish our images to the
ASF Docker Hub

-        Use Jenkins only to build and publish the Maven artifacts to the
ASF Nexus and some other CI pipeline outside of Jenkins (Travis or Github
Actions) to push to an apache-streampipes account on Docker Hub


What do you think? Maybe there's also a better option I'm overlooking ;-)




  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message