From commits-return-1218-archive-asf-public=cust-asf.ponee.io@yetus.apache.org Thu Jan 3 08:48:27 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 65070180608 for ; Thu, 3 Jan 2019 08:48:26 +0100 (CET) Received: (qmail 88488 invoked by uid 500); 3 Jan 2019 07:48:25 -0000 Mailing-List: contact commits-help@yetus.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@yetus.apache.org Delivered-To: mailing list commits@yetus.apache.org Received: (qmail 88479 invoked by uid 99); 3 Jan 2019 07:48:25 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2019 07:48:25 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id CAFBE821AE; Thu, 3 Jan 2019 07:48:24 +0000 (UTC) Date: Thu, 03 Jan 2019 07:48:24 +0000 To: "commits@yetus.apache.org" Subject: [yetus] branch master updated: YETUS-683. Publish apache/yetus docker images on hub.docker.com MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <154650170477.24561.18444396554031669014@gitbox.apache.org> From: aw@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: yetus X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: 28ef852e2685e845ae20528e2ec872f2599e8e76 X-Git-Newrev: 8c3c818285ee4e03bf31c56f678924ca37cfaf6e X-Git-Rev: 8c3c818285ee4e03bf31c56f678924ca37cfaf6e X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. aw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yetus.git The following commit(s) were added to refs/heads/master by this push: new 8c3c818 YETUS-683. Publish apache/yetus docker images on hub.docker.com 8c3c818 is described below commit 8c3c818285ee4e03bf31c56f678924ca37cfaf6e Author: Allen Wittenauer AuthorDate: Sun Nov 11 22:26:26 2018 -0800 YETUS-683. Publish apache/yetus docker images on hub.docker.com Signed-off-by: Allen Wittenauer --- .../documentation/in-progress/precommit-basic.md | 6 +- .../documentation/in-progress/precommit-robots.md | 59 ++++++++++++++- asf-site-src/source/downloads.html.md.erb | 2 +- asf-site-src/source/yetus-docker-image.md | 86 ++++++++++++++++++++++ 4 files changed, 147 insertions(+), 6 deletions(-) diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md index 5e24d28..63d25f2 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-basic.md +++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md @@ -84,7 +84,7 @@ Build Tools: Automation and Isolation: * [Circle CI](https://www.circleci.com) -* [Docker](https://www.docker.com) version 1.6.0+ +* [Docker](https://www.docker.com) version 1.7.0+ * [Gitlab CI](https://www.gitlab.com) * [Jenkins](https://www.jenkins-ci.org) * [Travis CI](https://www.travis-ci.com) @@ -329,10 +329,10 @@ NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode. If JA `test-patch` also has a mode to utilize Docker: ```bash -$ test-patch. (other options) --docker +$ test-patch (other options) --docker ``` -This will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile and other Docker-specific features, see [precommit-docker](../precommit-docker) for more information. +This command will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile and other Docker-specific features, see the specific [precommit Docker support](../precommit-docker) page and the [Apache Yetus Docker Hub Images](/yetus-docker-image) page for more information on the convenience Docker images. # In Closing diff --git a/asf-site-src/source/documentation/in-progress/precommit-robots.md b/asf-site-src/source/documentation/in-progress/precommit-robots.md index b0fab18..faa487c 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-robots.md +++ b/asf-site-src/source/documentation/in-progress/precommit-robots.md @@ -38,6 +38,23 @@ TRIGGER: ${CIRCLECI}=true Circle CI support in `test-patch` is limited to github.com. Artifacts (the `--patch-dir` directory) location needs to be handled set on the command line. Linking to the logs is not currently supported. +To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `.circleci/config.yaml` file, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile: + +```yaml +jobs: + build: + docker: + - image: apache/yetus:0.9.0 + + environment: + JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + + ... +``` + +See also + * See also the source tree's `.circleci/config.yaml` for some tips and tricks. + Gitlab CI ========= @@ -54,6 +71,21 @@ Artifacts, patch logs, etc are configured to go to a yetus-out directory in the ``` +To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `.gitlab-ci.yml` file, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile: + +```yaml +job: + image: apache/yetus:0.9.0 + allow_failure: true + variables: + JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + + ... +``` + +See also + * See also the source tree's `.gitlab-ci.yml` for some tips and tricks. + Jenkins ======= @@ -66,9 +98,29 @@ There is some support for a few well known environment variables: * `${GIT_URL}` will trigger the same extra handling if 'github' or 'gitlab' appear in the string. * If `${ghprbPullId}` is set, then test-patch will configure itself for a Github-style PR. +To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `Jenkinsfile`, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile: + +```groovy +pipeline { + agent { + docker { + image 'apache/yetus:0.9.0' + args '-v /var/run/docker.sock:/var/run/docker.sock' + } + } + + environment { + JAVA_HOME = '/usr/lib/jvm/java-8-openjdk-amd64' + } + + ... + +} + +``` See also - * See also the source tree's Jenkinsfile for some tips and tricks. + * See also the source tree's `Jenkinsfile` for some tips and tricks. * [precommit-admin](precommit-admin), for special utilities built for Jenkins. * [GitHub Branch Source Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin) * [GitHub Pull Request Builder Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin) @@ -83,9 +135,12 @@ Travis CI support will update the local checked out source repository to include If `${ARTIFACTS_PATH}` is configured, then `--patch-dir` is set to the first listed directory path. However, links to the location logs must still be configured manually. -Personalities will override the auto-detected Github repository information. It may be necessary to manually configure it in your .travis.yml file. +Personalities will override the auto-detected Github repository information. It may be necessary to manually configure it in your `.travis.yml` file. +As of this writing, it is not possible to make the Travis CI build environment use the Apache Yetus pre-built docker images without using ` docker run` in the before_install phase. Therefore, using the image is the same as described in the [Apache Yetus Docker Hub Images](/yetus-docker-image) page. +See also + * See also the source tree's `.travis.yml` for some tips and tricks. Manual Configuration ==================== diff --git a/asf-site-src/source/downloads.html.md.erb b/asf-site-src/source/downloads.html.md.erb index e3c0f11..27c362e 100644 --- a/asf-site-src/source/downloads.html.md.erb +++ b/asf-site-src/source/downloads.html.md.erb @@ -31,7 +31,7 @@ should verify the integrity of these artifacts via [the ASF procedure](https://w using the [KEYS file for our project](https://dist.apache.org/repos/dist/release/yetus/KEYS). In addition to the artifacts below, we publish convenience artifacts in some -third-party repositories (e.g. Maven Central). +third-party repositories (e.g. Maven Central) and [Docker Hub](../yetus-docker-image). ## Releases diff --git a/asf-site-src/source/yetus-docker-image.md b/asf-site-src/source/yetus-docker-image.md new file mode 100644 index 0000000..963cbf5 --- /dev/null +++ b/asf-site-src/source/yetus-docker-image.md @@ -0,0 +1,86 @@ + + +Convenience Docker Hub Images +=========================== + +While not official releases, the Apache Yetus project makes available two types of images on hub.docker.com: + + * apache/yetus-base + + This image is the same as the 'built-in' Dockerfile when `--docker` is used without `--dockerfile` on the precommit command line. It includes all of the pre-requisites as needed by the various Apache Yetus components. It is located in `precommit/src/main/shell/test-patch-docker/`. + + * apache/yetus + + This image is the same as apache/yetus-base but includes a pre-built version of Apache Yetus as part of the base OS image. In other words, qbt, releasedocmaker, shelldocs, test-patch, etc., are in /usr/bin and available in the default path. It is generated from the Dockerfile located in the root of the source and is built with the options provided in the hooks directory. + +Both images should be suitable to be used as a building block or even directly if your build environment needs no other dependencies. These images are especially useful for various CI systems that require a Docker image to be used. + +Images are tagged such that 'master' represents the last successful Docker image build of the master branch. Images based off of the official source releases are tagged with a matching version number (e.g., 0.9.0). There is no 'latest' tagged image. It is recommended that users choose a stable tag so as not to be surprised by incompatible changes. + +File Access +======== + +All of the executables that Apache Yetus provides requires access to one or more directories. These directories should be provided on the command line via the volume flags to docker run. For example, to run shelldocs against test-patch.sh: + +```bash +docker run \ + --rm \ + -v /tmp/out:/output \ + -v /src/precommit/src/main/shell:/input:ro \ + apache/yetus:0.9.0 \ + shelldocs --output /output/test-patch.md --input /input/test-patch.sh +``` + +In this example, we mount two volumes: one for input (marked read-only), one for output. After execution, shelldocs has processed test-patch.sh and generated a test-patch.md file in /tmp/out on our local machine. + +Precommit also works, assuming that Apache Yetus image has all of your project's dependencies. For example: + +```bash +docker run \ + --rm \ + -v /tmp/patchdir:/patchdir \ + -v /src:/basedir \ + apache/yetus:0.9.0 \ + qbt \ + --plugins=all \ + --basedir=/basedir \ + --patch-dir=/patchdir \ + --project=yetus +``` +If your project needs additional dependencies, it is trivial to build off of the Apache Yetus image: + +```Dockerfile +FROM apache/yetus:0.9.0 +RUN apt-get -q update && apt-get -q install -y \ + clang \ + libssl-dev \ + valgrind \ + zlib1g-dev +``` + ```bash +docker build -t project/build:0.9.0 -f . +``` + +This example builds a docker image based off of Apache Yetus 0.9.0 but with the additions of clang, some development libraries, and valgrind. Now project/build:0.9.0 can be used instead of apache/yetus:0.9.0 since it has all of Apache Yetus and the additions needed by our project. + +A Note About Precommit +================ + +test-patch and friends have direct support for Docker outside of the convenience images. That information is covered in-depth in the[precommit-docker](../precommit-docker) section.