From dev-return-105-archive-asf-public=cust-asf.ponee.io@streampipes.apache.org Thu Dec 12 15:16:39 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 50B0018061A for ; Thu, 12 Dec 2019 16:16:39 +0100 (CET) Received: (qmail 96490 invoked by uid 500); 12 Dec 2019 15:16:38 -0000 Mailing-List: contact dev-help@streampipes.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@streampipes.apache.org Delivered-To: mailing list dev@streampipes.apache.org Received: (qmail 96478 invoked by uid 99); 12 Dec 2019 15:16:38 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Dec 2019 15:16:38 +0000 Received: from ipe-husky.fzi.de (ipe-husky.fzi.de [141.21.42.75]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 3E43110FC for ; Thu, 12 Dec 2019 15:16:38 +0000 (UTC) From: Philipp Zehnder Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3601.0.10\)) Subject: Re: CI Pipelines Date: Thu, 12 Dec 2019 16:16:36 +0100 References: <011501d5b06d$b34476b0$19cd6410$@apache.org> To: dev@streampipes.apache.org In-Reply-To: <011501d5b06d$b34476b0$19cd6410$@apache.org> Message-Id: <22CEFA79-F2E1-4160-A0F8-4BB008366002@apache.org> X-Mailer: Apple Mail (2.3601.0.10) Hi Dominik, I would also prefer to have a separate Docker Hub Repo. Regarding the build process: Would it be possible use Jenkins to push the SNAPSHOTs into Nexus (for = PE developers) and a separate build pipeline with for example with = GitHub actions which first builds the .jar files and then the containers = without pulling or pushing to Nexus? For Releases we could have a separate build pipeline which we can = trigger manually to build =E2=80=98release docker images=E2=80=99 = directly with the artifacts from Nexus. Cheers, Philipp > On 11. Dec 2019, at 22:55, Dominik Riemer wrote: >=20 > Hi, >=20 > I'm currently trying to figure out what could be the best CI strategy = in the > ASF infrastructure. >=20 >=20 >=20 > For StreamPipes development, the following things are important for a = good > development experience: >=20 > - Users with different roles (e.g., UI dev, backend dev, = pipeline > element dev) should be able to work independently on components = without > needing to setup the whole development stack >=20 > - We currently manage this in Docker, e.g., when you are = developing > on the UI, all other components and services run in Docker, so that UI > developers don't need to care about how to setup a Maven project and = vice > versa >=20 > - When developing pipeline elements (which are in the repo > incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes > artifacts need to be fetched from a Maven repository, so that pipeline > element development can happen based on the latest SNAPSHOT version >=20 >=20 >=20 > Currently, we use a build pipeline in Gitlab CI as follows (taking the > example of the core at incubator-streampipes): >=20 > - Build Maven artifacts >=20 > - Build UI >=20 > - Build Docker Images and deploy these images to Docker Hub=20 >=20 > - Deploy Maven artifacts to Nexus >=20 > This was quite a smooth and integrated process. >=20 >=20 >=20 > So after having talked to Chris and INFRA, it seems that only Jenkins = is > allowed to publish to the Apache Maven Nexus, but Jenkins can only = push to > the official Apache Docker Hub account. As our development is heavily > focused around Docker (we are building around 20 Docker Images for the > individual services), I'd personally prefer to use the 2nd way = proposed in > the Docker Hub release policy ( > = s> > = https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelin= es > , e.g., having an apache-streampipes organization account on Docker = Hub) as > it gives users a much better overview on what images are currently > available. However, this won't work from Jenkins as it's not possible = to > push to a non-ASF-namespace. >=20 >=20 >=20 > I played around a little bit with Jenkins pipelines and while = Docker-based > builds are generally supported, I'm not yet 100% sure how easy it is = to > share artifacts over stages. >=20 >=20 >=20 > So I guess we have two options, both approaches have pros and cons: >=20 > - Use Jenkins for the complete pipeline and publish our images = to the > ASF Docker Hub >=20 > - Use Jenkins only to build and publish the Maven artifacts to = the > ASF Nexus and some other CI pipeline outside of Jenkins (Travis or = Github > Actions) to push to an apache-streampipes account on Docker Hub >=20 >=20 >=20 > What do you think? Maybe there's also a better option I'm overlooking = ;-) >=20 >=20 >=20 > Dominik >=20 >=20 >=20