streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominik Riemer" <rie...@apache.org>
Subject RE: CI Pipelines
Date Thu, 12 Dec 2019 22:50:46 GMT
Hi Philipp,

yes, what you are proposing sounds good from my side.
We got an access token configured by Infra and I played a little bit with Github Actions and
it's great fun. I also asked Infra to setup Nexus, so I'd propose to try the approach you
were outlining thereafter.

Concerning your last point, AFAIK we shouldn't do releases from a CI pipeline as we did in
the pre-ASF era. Instead, a release manager from the community would lead the preparation
of the release, call the vote and is also responsible for pushing convenience libraries to
our channels after a successful vote. I hope that Justin will comment in case I'm wrong here.

Dominik

-----Original Message-----
From: Philipp Zehnder <zehnder@apache.org> 
Sent: Thursday, December 12, 2019 4:17 PM
To: dev@streampipes.apache.org
Subject: Re: CI Pipelines

Hi Dominik,

I would also prefer to have a separate Docker Hub Repo.

Regarding the build process:

Would it be possible use Jenkins to push the SNAPSHOTs into Nexus (for PE developers) and
a separate build pipeline with for example with GitHub actions which first builds the .jar
files and then the containers without pulling or pushing to Nexus?

For Releases we could have a separate build pipeline which we can trigger manually to build
‘release docker images’ directly with the artifacts from Nexus.

Cheers,
Philipp

> On 11. Dec 2019, at 22:55, Dominik Riemer <riemer@apache.org> wrote:
> 
> Hi,
> 
> I'm currently trying to figure out what could be the best CI strategy 
> in the ASF infrastructure.
> 
> 
> 
> For StreamPipes development, the following things are important for a 
> good development experience:
> 
> -        Users with different roles (e.g., UI dev, backend dev, pipeline
> element dev) should be able to work independently on components 
> without needing to setup the whole development stack
> 
> -        We currently manage this in Docker, e.g., when you are developing
> on the UI, all other components and services run in Docker, so that UI 
> developers don't need to care about how to setup a Maven project and 
> vice versa
> 
> -        When developing pipeline elements (which are in the repo
> incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes 
> artifacts need to be fetched from a Maven repository, so that pipeline 
> element development can happen based on the latest SNAPSHOT version
> 
> 
> 
> Currently, we use a build pipeline in Gitlab CI as follows (taking the 
> example of the core at incubator-streampipes):
> 
> -        Build Maven artifacts
> 
> -        Build UI
> 
> -        Build Docker Images and deploy these images to Docker Hub 
> 
> -        Deploy Maven artifacts to Nexus
> 
> This was quite a smooth and integrated process.
> 
> 
> 
> So after having talked to Chris and INFRA, it seems that only Jenkins 
> is allowed to publish to the Apache Maven Nexus, but Jenkins can only 
> push to the official Apache Docker Hub account. As our development is 
> heavily focused around Docker (we are building around 20 Docker Images 
> for the individual services), I'd personally prefer to use the 2nd way 
> proposed in the Docker Hub release policy ( 
> <https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGui
> deline
> s>
> https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuid
> elines , e.g., having an apache-streampipes organization account on 
> Docker Hub) as it gives users a much better overview on what images 
> are currently available. However, this won't work from Jenkins as it's 
> not possible to push to a non-ASF-namespace.
> 
> 
> 
> I played around a little bit with Jenkins pipelines and while 
> Docker-based builds are generally supported, I'm not yet 100% sure how 
> easy it is to share artifacts over stages.
> 
> 
> 
> So I guess we have two options, both approaches have pros and cons:
> 
> -        Use Jenkins for the complete pipeline and publish our images to the
> ASF Docker Hub
> 
> -        Use Jenkins only to build and publish the Maven artifacts to the
> ASF Nexus and some other CI pipeline outside of Jenkins (Travis or 
> Github
> Actions) to push to an apache-streampipes account on Docker Hub
> 
> 
> 
> What do you think? Maybe there's also a better option I'm overlooking 
> ;-)
> 
> 
> 
> Dominik
> 
> 
> 



Mime
View raw message