flume-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mpe...@apache.org
Subject flume git commit: Change how we build and sign release artifacts
Date Sun, 29 Jan 2017 00:50:09 GMT
Repository: flume
Updated Branches:
  refs/heads/trunk a84491472 -> 18453d3ef


Change how we build and sign release artifacts

These simple scripts automate creation and signing of the release
artifacts. They help guarantee that the source artifacts published match
the Git tag.

Updated the How To Release documentation to explain how to use the
scripts.

This closes #98

Reviewers: Denes Arvay, Bessenyei Balázs Donát


Project: http://git-wip-us.apache.org/repos/asf/flume/repo
Commit: http://git-wip-us.apache.org/repos/asf/flume/commit/18453d3e
Tree: http://git-wip-us.apache.org/repos/asf/flume/tree/18453d3e
Diff: http://git-wip-us.apache.org/repos/asf/flume/diff/18453d3e

Branch: refs/heads/trunk
Commit: 18453d3ef182e92785d9fb74ff4fbdc244a4bc88
Parents: a844914
Author: Mike Percy <mpercy@apache.org>
Authored: Wed Oct 12 18:59:58 2016 +0200
Committer: Mike Percy <mpercy@apache.org>
Committed: Sat Jan 28 16:49:35 2017 -0800

----------------------------------------------------------------------
 dev-docs/HowToRelease.md               | 79 ++++++++++++++++-------------
 dev-support/create-source-archive.sh   | 51 +++++++++++++++++++
 dev-support/generate-source-release.sh | 49 ++++++++++++++++++
 dev-support/includes.sh                | 40 +++++++++++++++
 dev-support/sign-checksum-artifact.sh  | 46 +++++++++++++++++
 5 files changed, 231 insertions(+), 34 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flume/blob/18453d3e/dev-docs/HowToRelease.md
----------------------------------------------------------------------
diff --git a/dev-docs/HowToRelease.md b/dev-docs/HowToRelease.md
index be650b2..388b73d 100644
--- a/dev-docs/HowToRelease.md
+++ b/dev-docs/HowToRelease.md
@@ -229,30 +229,52 @@ If an rc2, rc3 etc is needed, simply create a new rc tag:
     git push origin release-X.Y.Z-rc2
 
 
-### Performing sanity check
+### Preparing to sign the artifacts
+
+All artifacts must be signed and checksummed. In order to sign a release
+you will need a PGP key. You should get your key signed by a few other
+people. You will also need to recv their keys from a public key server.
+See the [Apache release signing](https://www.apache.org/dev/release-signing)
+page for more details.
+
+1\. Add your key to the
+[KEYS](https://dist.apache.org/repos/dist/release/flume/KEYS) file:
+
+
+    (gpg --list-sigs <your-email> && gpg --armor --export <your-email>)
>> KEYS
+
+
+And commit the changes.
 
-1\. Check out the candidate tag
 
+### Generating and signing the source artifacts
 
-    git checkout release-X.Y.Z-rc1
+There is a script in the Flume source tree for generating and signing the Flume
+source artifacts. Once the release candidate is tagged, generate the source
+release with the following steps.
 
+1\. From the top of the Flume source tree, create a directory for the artifacts
+    and then generate them:
 
-2\. Generate a tarball
 
+    mkdir ./source-artifacts
+    ./dev-support/generate-source-release.sh X.Y.Z release-X.Y.Z-rc1 ./source-artifacts/
 
-    mvn clean install -DskipTests
 
+The artifacts will be placed in the directory you specify (in this case,
+`./source-artifacts`)
 
-3\. Unpack the source tarball
+
+### Testing the source tarball
+
+1\. Unpack the source tarball
 
 
-    cd flume-ng-dist/target
-    rm -rf ./apache-flume-X.Y.Z-src/
     tar xzvf apache-flume-X.Y.Z-src.tar.gz
 
 
-4\. Do another full build inside the source tarball. This time, allow all
-unit tests & integration tests to run and also include the docs
+2\. Do a full build inside the source tarball. Allow all unit tests &
+integration tests to run and also include the docs.
 
 
     cd apache-flume-X.Y.Z-src
@@ -260,29 +282,17 @@ unit tests & integration tests to run and also include the docs
     mvn clean install -Psite -DskipTests
 
 
-5\. Verify that the HTML docs that should have been generated inside the
+3\. Verify that the HTML docs that should have been generated inside the
 binary artifact under /docs are there and do not have rendering errors.
 
-### Signatures and Checksums
 
-All artifacts must be signed and checksummed. In order to sign a release
-you will need a PGP key. You should get your key signed by a few other
-people. You will also need to recv their keys from a public key server.
-See the [Apache release
-signing](https://www.apache.org/dev/release-signing)
-page for more details.
+### Generating, signing, and deploying the binary artifacts
 
-1\. Add your key to the
-[KEYS](https://dist.apache.org/repos/dist/release/flume/KEYS)
-file:
+Maven is configured to generate, sign, and deploy the binary artifacts
+automatically. Use the following steps to do that:
 
 
-    (gpg --list-sigs <your-email> && gpg --armor --export <your-email>)
>> KEYS
-
-
-And commit the changes.
-
-2\. Create and sign the artifacts, including site docs. This pushes the
+1\. Create and sign the artifacts, including site docs. This pushes the
 signed artifacts to the ASF staging repository.
 
 In order to do this, you will need a settings.xml file with your
@@ -301,14 +311,14 @@ placed in \~/.m2/settings.xml and might look something like this:
     </settings>
 
 
-Once your settings.xml file is correct, you run the following from the
-flume root directory to generate and deploy the artifacts:
+2\. Once your settings.xml file is correct, run the following from the
+Flume source directory to generate and deploy the artifacts:
 
 
     mvn clean deploy -Psite -Psign -DskipTests
 
 
-This will sign, hash, and upload each artifact to Nexus.
+This will sign, checksum, and upload each artifact to Nexus.
 
 Note: the checksum files will not be mirrored; They should be downloaded
 from the main apache dist site.
@@ -329,16 +339,17 @@ OK, and then click Close using "Apache Flume X.Y.Z" as the description
 to allow others to see the repository. Note that the staging repository
 will have a numeric id associated with it that will be used later
 
-4\. Copy artifacts to people.apache.org
+4\. Copy the source artifacts you built locally to people.apache.org
+
+    $ rsync -e ssh -av source-artifacts/apache-flume-X.Y.Z-src.tar.gz* people.apache.org:public_html/apache-flume-X-Y.Z-rcN/
 
-Copy the apache-flume-X.Y.Z-{bin,src}.tar.gz{,.{asc,md5,sha1}} files to
-people.apache.org.
+5\. Copy the binary artifacts you deployed via Maven to people.apache.org
 
     $ ssh people.apache.org
     $ cd public_html
     $ mkdir apache-flume-X.Y.Z-rcN
     $ cd apache-flume-X.Y.Z-rcN
-    $ wget --no-check-certificate https://repository.apache.org/content/repositories/orgapacheflume-XXXX/org/apache/flume/flume-ng-dist/X.Y.Z/flume-ng-dist-X.Y.Z-{src,bin}.tar.gz{,.{asc,md5,sha1}}
+    $ wget --no-check-certificate https://repository.apache.org/content/repositories/orgapacheflume-XXXX/org/apache/flume/flume-ng-dist/X.Y.Z/flume-ng-dist-X.Y.Z-bin.tar.gz{,.{asc,md5,sha1}}
     $ for file in flume-ng-dist-*; do mv $file $(echo $file | sed -e "s/flume-ng-dist/apache-flume/g");done
 
 

http://git-wip-us.apache.org/repos/asf/flume/blob/18453d3e/dev-support/create-source-archive.sh
----------------------------------------------------------------------
diff --git a/dev-support/create-source-archive.sh b/dev-support/create-source-archive.sh
new file mode 100755
index 0000000..a938b44
--- /dev/null
+++ b/dev-support/create-source-archive.sh
@@ -0,0 +1,51 @@
+#!/bin/bash -e
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+################################################################################
+# Script to generate a source release tarball.
+# The contract of this script is that it outputs the location of the generated
+# tarball to stdout upon successful completion.
+################################################################################
+DEV_SUPPORT=$(cd $(dirname $0); pwd)
+source "$DEV_SUPPORT/includes.sh"
+
+VERSION_NUMBER=$1
+GIT_TAG=$2
+OUTPUT_DIR=$3
+
+if [[ -z "$VERSION_NUMBER" || -z "$GIT_TAG" || -z "$OUTPUT_DIR" ]]; then
+  echo "Usage: $0 VERSION_NUMBER GIT_TAG OUTPUT_DIR" 1>&2
+  echo "Example: $0 1.7.0 release-1.7.0-rc1 target" 1>&2
+  exit 1
+fi
+
+[ ! -d "$OUTPUT_DIR" ] && error "Output directory $OUTPUT_DIR does not exist."
+ABS_OUTPUT_DIR=$(cd $OUTPUT_DIR; pwd)
+
+EXT=tar.gz
+ARTIFACT_NAME=apache-flume-${VERSION_NUMBER}-src
+ARTIFACT_PATH=$ABS_OUTPUT_DIR/$ARTIFACT_NAME.$EXT
+
+# Need to call git archive from the root of the tree.
+cd $DEV_SUPPORT/..
+
+echo git archive --prefix=$ARTIFACT_NAME/ --output=$ARTIFACT_PATH --format "$EXT" "$GIT_TAG"
1>&2
+git archive --prefix=$ARTIFACT_NAME/ --output=$ARTIFACT_PATH --format "$EXT" "$GIT_TAG"
+
+echo $ARTIFACT_PATH
+exit 0

http://git-wip-us.apache.org/repos/asf/flume/blob/18453d3e/dev-support/generate-source-release.sh
----------------------------------------------------------------------
diff --git a/dev-support/generate-source-release.sh b/dev-support/generate-source-release.sh
new file mode 100755
index 0000000..029d6c4
--- /dev/null
+++ b/dev-support/generate-source-release.sh
@@ -0,0 +1,49 @@
+#!/bin/bash -e
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+################################################################################
+# Script to generate and sign a source release.
+################################################################################
+DEV_SUPPORT=$(cd $(dirname $0); pwd)
+source "$DEV_SUPPORT/includes.sh"
+
+VERSION_NUMBER=$1
+GIT_TAG=$2
+OUTPUT_DIR=$3
+
+if [[ -z "$VERSION_NUMBER" || -z "$GIT_TAG" || -z "$OUTPUT_DIR" ]]; then
+  echo "Usage: $0 VERSION_NUMBER GIT_TAG OUTPUT_DIR"
+  echo "Example: $0 1.7.0 release-1.7.0-rc1 target"
+  exit 1
+fi
+
+# Generate the source artifact.
+echo "Creating source archive..."
+CREATE_ARCHIVE=$DEV_SUPPORT/create-source-archive.sh
+ARCHIVE_PATH=$($CREATE_ARCHIVE "$VERSION_NUMBER" "$GIT_TAG" "$OUTPUT_DIR")
+[ $? != 0 ] && error "Failed to generate source archive. $CREATE_ARCHIVE returned
$?"
+[ ! -r $ARCHIVE_PATH ] && error "Failed to generate source archive. Unknown error."
+
+# Sign and checksum the source artifact.
+echo "Signing source artifact..."
+SIGN_ARTIFACT=$DEV_SUPPORT/sign-checksum-artifact.sh
+$SIGN_ARTIFACT "$ARCHIVE_PATH"
+
+echo "Release artifacts generated in $OUTPUT_DIR"
+exit 0
+

http://git-wip-us.apache.org/repos/asf/flume/blob/18453d3e/dev-support/includes.sh
----------------------------------------------------------------------
diff --git a/dev-support/includes.sh b/dev-support/includes.sh
new file mode 100644
index 0000000..255a499
--- /dev/null
+++ b/dev-support/includes.sh
@@ -0,0 +1,40 @@
+#!/bin/bash -e
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+################################################################################
+# Utilities for dev-support scripts.
+################################################################################
+
+# Print an error message and exit.
+error() {
+  echo $1 1>&2
+  exit 1
+}
+
+# Searches the PATH for each command name passed, and returns the path of the
+# first one found.
+find_in_path() {
+  for COMMAND in "$@"; do
+    FOUND=$(which $COMMAND)
+    if [ -n "$FOUND" ]; then
+      echo "$FOUND"
+      return
+    fi
+  done
+  error "Cannot find $1. Please install $1 to continue."
+}

http://git-wip-us.apache.org/repos/asf/flume/blob/18453d3e/dev-support/sign-checksum-artifact.sh
----------------------------------------------------------------------
diff --git a/dev-support/sign-checksum-artifact.sh b/dev-support/sign-checksum-artifact.sh
new file mode 100755
index 0000000..253c7df
--- /dev/null
+++ b/dev-support/sign-checksum-artifact.sh
@@ -0,0 +1,46 @@
+#!/bin/bash -e
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+################################################################################
+# Sign and checksum release artifacts.
+################################################################################
+DEV_SUPPORT=$(cd $(dirname $0); pwd)
+source "$DEV_SUPPORT/includes.sh"
+
+usage() {
+  echo "Usage: $0 RELEASE_ARTIFACT" 1>&2
+  echo "Example: $0 ./apache-flume-1.7.0-src.tar.gz" 1>&2
+  exit 1
+}
+
+ARTIFACT=$1
+if [ ! -r "$ARTIFACT" ]; then
+  echo "The artifact at $ARTIFACT does not exist or is not readable." 1>&2
+  usage
+fi
+
+# The tools we need.
+GPG=$(find_in_path gpg)
+MD5=$(find_in_path md5sum md5)
+SHA1=$(find_in_path sha1sum shasum)
+
+# Now sign and checksum the artifact.
+set -x
+$GPG --sign $ARTIFACT
+$MD5 < $ARTIFACT > $ARTIFACT.md5
+$SHA1 < $ARTIFACT > $ARTIFACT.sha1


Mime
View raw message