From dev-return-100-archive-asf-public=cust-asf.ponee.io@streampipes.apache.org Wed Dec 11 21:55:29 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 45E84180638 for ; Wed, 11 Dec 2019 22:55:29 +0100 (CET) Received: (qmail 87502 invoked by uid 500); 11 Dec 2019 21:55:28 -0000 Mailing-List: contact dev-help@streampipes.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@streampipes.apache.org Delivered-To: mailing list dev@streampipes.apache.org Received: (qmail 87490 invoked by uid 99); 11 Dec 2019 21:55:28 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2019 21:55:28 +0000 Received: from ipeente (HSI-KBW-46-223-163-192.hsi.kabel-badenwuerttemberg.de [46.223.163.192]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 17AD14FB3 for ; Wed, 11 Dec 2019 21:55:27 +0000 (UTC) From: "Dominik Riemer" To: Subject: CI Pipelines Date: Wed, 11 Dec 2019 22:55:26 +0100 Message-ID: <011501d5b06d$b34476b0$19cd6410$@apache.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0116_01D5B076.150AB370" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AdWwauS2k913gej/ReaEXkNIDihJzQ== Content-Language: de ------=_NextPart_000_0116_01D5B076.150AB370 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi, I'm currently trying to figure out what could be the best CI strategy in the ASF infrastructure. For StreamPipes development, the following things are important for a good development experience: - Users with different roles (e.g., UI dev, backend dev, pipeline element dev) should be able to work independently on components without needing to setup the whole development stack - We currently manage this in Docker, e.g., when you are developing on the UI, all other components and services run in Docker, so that UI developers don't need to care about how to setup a Maven project and vice versa - When developing pipeline elements (which are in the repo incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes artifacts need to be fetched from a Maven repository, so that pipeline element development can happen based on the latest SNAPSHOT version Currently, we use a build pipeline in Gitlab CI as follows (taking the example of the core at incubator-streampipes): - Build Maven artifacts - Build UI - Build Docker Images and deploy these images to Docker Hub - Deploy Maven artifacts to Nexus This was quite a smooth and integrated process. So after having talked to Chris and INFRA, it seems that only Jenkins is allowed to publish to the Apache Maven Nexus, but Jenkins can only push to the official Apache Docker Hub account. As our development is heavily focused around Docker (we are building around 20 Docker Images for the individual services), I'd personally prefer to use the 2nd way proposed in the Docker Hub release policy ( https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines , e.g., having an apache-streampipes organization account on Docker Hub) as it gives users a much better overview on what images are currently available. However, this won't work from Jenkins as it's not possible to push to a non-ASF-namespace. I played around a little bit with Jenkins pipelines and while Docker-based builds are generally supported, I'm not yet 100% sure how easy it is to share artifacts over stages. So I guess we have two options, both approaches have pros and cons: - Use Jenkins for the complete pipeline and publish our images to the ASF Docker Hub - Use Jenkins only to build and publish the Maven artifacts to the ASF Nexus and some other CI pipeline outside of Jenkins (Travis or Github Actions) to push to an apache-streampipes account on Docker Hub What do you think? Maybe there's also a better option I'm overlooking ;-) Dominik ------=_NextPart_000_0116_01D5B076.150AB370--