streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philipp Zehnder <zehn...@apache.org>
Subject Re: CI Pipelines
Date Thu, 12 Dec 2019 15:16:36 GMT
Hi Dominik,

I would also prefer to have a separate Docker Hub Repo.

Regarding the build process:

Would it be possible use Jenkins to push the SNAPSHOTs into Nexus (for PE developers) and
a separate build pipeline with for example with GitHub actions which first builds the .jar
files and then the containers without pulling or pushing to Nexus?

For Releases we could have a separate build pipeline which we can trigger manually to build
‘release docker images’ directly with the artifacts from Nexus.

Cheers,
Philipp

> On 11. Dec 2019, at 22:55, Dominik Riemer <riemer@apache.org> wrote:
> 
> Hi,
> 
> I'm currently trying to figure out what could be the best CI strategy in the
> ASF infrastructure.
> 
> 
> 
> For StreamPipes development, the following things are important for a good
> development experience:
> 
> -        Users with different roles (e.g., UI dev, backend dev, pipeline
> element dev) should be able to work independently on components without
> needing to setup the whole development stack
> 
> -        We currently manage this in Docker, e.g., when you are developing
> on the UI, all other components and services run in Docker, so that UI
> developers don't need to care about how to setup a Maven project and vice
> versa
> 
> -        When developing pipeline elements (which are in the repo
> incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes
> artifacts need to be fetched from a Maven repository, so that pipeline
> element development can happen based on the latest SNAPSHOT version
> 
> 
> 
> Currently, we use a build pipeline in Gitlab CI as follows (taking the
> example of the core at incubator-streampipes):
> 
> -        Build Maven artifacts
> 
> -        Build UI
> 
> -        Build Docker Images and deploy these images to Docker Hub 
> 
> -        Deploy Maven artifacts to Nexus
> 
> This was quite a smooth and integrated process.
> 
> 
> 
> So after having talked to Chris and INFRA, it seems that only Jenkins is
> allowed to publish to the Apache Maven Nexus, but Jenkins can only push to
> the official Apache Docker Hub account. As our development is heavily
> focused around Docker (we are building around 20 Docker Images for the
> individual services), I'd personally prefer to use the 2nd way proposed in
> the Docker Hub release policy (
> <https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuideline
> s>
> https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines
> , e.g., having an apache-streampipes organization account on Docker Hub) as
> it gives users a much better overview on what images are currently
> available. However, this won't work from Jenkins as it's not possible to
> push to a non-ASF-namespace.
> 
> 
> 
> I played around a little bit with Jenkins pipelines and while Docker-based
> builds are generally supported, I'm not yet 100% sure how easy it is to
> share artifacts over stages.
> 
> 
> 
> So I guess we have two options, both approaches have pros and cons:
> 
> -        Use Jenkins for the complete pipeline and publish our images to the
> ASF Docker Hub
> 
> -        Use Jenkins only to build and publish the Maven artifacts to the
> ASF Nexus and some other CI pipeline outside of Jenkins (Travis or Github
> Actions) to push to an apache-streampipes account on Docker Hub
> 
> 
> 
> What do you think? Maybe there's also a better option I'm overlooking ;-)
> 
> 
> 
> Dominik
> 
> 
> 


Mime
View raw message