Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A13C718D29 for ; Fri, 4 Mar 2016 04:59:12 +0000 (UTC) Received: (qmail 62025 invoked by uid 500); 4 Mar 2016 04:59:12 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 61582 invoked by uid 500); 4 Mar 2016 04:59:12 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 61490 invoked by uid 99); 4 Mar 2016 04:59:12 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2016 04:59:12 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E36BFDFA42; Fri, 4 Mar 2016 04:59:11 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: wang@apache.org To: common-commits@hadoop.apache.org Date: Fri, 04 Mar 2016 04:59:13 -0000 Message-Id: <142c230fa8eb4ada95d1be0caada7380@git.apache.org> In-Reply-To: <295fd34804cf4c928fba4426128f7c65@git.apache.org> References: <295fd34804cf4c928fba4426128f7c65@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [3/3] hadoop git commit: HADOOP-12651. Replace dev-support with wrappers to Yetus (aw) HADOOP-12651. Replace dev-support with wrappers to Yetus (aw) (cherry picked from commit 2a867355dfce880bb82257508862ec26bc7f16b9) (cherry picked from commit f74ecffaa6f01719dbb9a1f5e5704f85715ecf70) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e2ddc9a4 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e2ddc9a4 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e2ddc9a4 Branch: refs/heads/branch-2.8 Commit: e2ddc9a469aab21f15209268d90f2fe2648263d1 Parents: a9f041d Author: Allen Wittenauer Authored: Thu Jan 21 13:03:34 2016 -0800 Committer: Andrew Wang Committed: Thu Mar 3 20:59:03 2016 -0800 ---------------------------------------------------------------------- dev-support/README.md | 57 +++++++++++ dev-support/bin/releasedocmaker | 18 ++++ dev-support/bin/shelldocs | 18 ++++ dev-support/bin/smart-apply-patch | 18 ++++ dev-support/bin/test-patch | 18 ++++ dev-support/bin/yetus-wrapper | 175 +++++++++++++++++++++++++++++++++ 6 files changed, 304 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/README.md ---------------------------------------------------------------------- diff --git a/dev-support/README.md b/dev-support/README.md new file mode 100644 index 0000000..b2e3476 --- /dev/null +++ b/dev-support/README.md @@ -0,0 +1,57 @@ + + +This directory contains tools to help in the development and release of Apache Hadoop. + +* bin + + * releasedocmaker + + Build release notes for a given Hadoop project or subproject. This is called from Maven when -Preleasedocs is used. See BUILDING.txt for more information. + + * shelldocs + + Create documentation for the Unix Shell API. This is called from Maven when -Pshelldocs is used. + + * smart-apply-patch + + Intelligently apply a patch file to a source tree. + + * test-patch + + Test a patch against a source tree. + +* create-release.sh + + Helps REs create a release of Apache Hadoop for distribution. + +* determine-flaky-tests-hadoop.py + + Given a jenkins test job, this script examines all runs of the job done within specified period of time (number of days prior to the execution time of this script), and reports all failed tests. + +* docker + + Various helpers for the start-build-env.sh script, including the Dockerfile itself. See parent BUILDING.txt for more information. + +* findHangingTest.sh + + Finds hanging test from Jenkins build output. + + +Previously, the scripts test-patch.sh, smart-apply-patch.sh, releasedocmaker.py, and shelldocs.py were in this directory. They have been moved to the Apache Yetus project (https://yetus.apache.org). These scripts have been replaced with wrapper scripts located in the bin directory. Command line options are generally different than the previous versions that shipped with older versions of Apache Hadoop. + +The wrapper scripts will download, verify (if GPG is installed), and cache a local copy of Apache Yetus in the hadoop/patchprocess directory. The version that is used may be overridden by setting the HADOOP\_YETUS\_VERSION environment variable. The cache directory may be overwritten by setting the HADOOP\_PATCHPROCESS directory. If a local version of Apache Yetus is already installed, it may be used instead by setting the YETUS\_HOME environment variable to point to that directory. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/bin/releasedocmaker ---------------------------------------------------------------------- diff --git a/dev-support/bin/releasedocmaker b/dev-support/bin/releasedocmaker new file mode 100755 index 0000000..16e4c7e --- /dev/null +++ b/dev-support/bin/releasedocmaker @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" releasedocmaker "$@" http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/bin/shelldocs ---------------------------------------------------------------------- diff --git a/dev-support/bin/shelldocs b/dev-support/bin/shelldocs new file mode 100755 index 0000000..18eda4d --- /dev/null +++ b/dev-support/bin/shelldocs @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" shelldocs "$@" http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/bin/smart-apply-patch ---------------------------------------------------------------------- diff --git a/dev-support/bin/smart-apply-patch b/dev-support/bin/smart-apply-patch new file mode 100755 index 0000000..3fd469f --- /dev/null +++ b/dev-support/bin/smart-apply-patch @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" smart-apply-patch --project=hadoop "$@" http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/bin/test-patch ---------------------------------------------------------------------- diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch new file mode 100755 index 0000000..8ff8119 --- /dev/null +++ b/dev-support/bin/test-patch @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop --skip-dir=dev-support "$@" http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2ddc9a4/dev-support/bin/yetus-wrapper ---------------------------------------------------------------------- diff --git a/dev-support/bin/yetus-wrapper b/dev-support/bin/yetus-wrapper new file mode 100755 index 0000000..37082d8 --- /dev/null +++ b/dev-support/bin/yetus-wrapper @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -o pipefail + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_error +{ + echo "$*" 1>&2 +} + +## @description Given a filename or dir, return the absolute version of it +## @audience public +## @stability stable +## @param directory +## @replaceable no +## @return 0 success +## @return 1 failure +## @return stdout abspath +function yetus_abs +{ + declare obj=$1 + declare dir + declare fn + + if [[ ! -e ${obj} ]]; then + return 1 + elif [[ -d ${obj} ]]; then + dir=${obj} + else + dir=$(dirname -- "${obj}") + fn=$(basename -- "${obj}") + fn="/${fn}" + fi + + dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P) + if [[ $? = 0 ]]; then + echo "${dir}${fn}" + return 0 + fi + return 1 +} + + +WANTED="$1" +shift +ARGV=("$@") + +HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.1.0} +BIN=$(yetus_abs "${BASH_SOURCE-$0}") +BINDIR=$(dirname "${BIN}") + +### +### if YETUS_HOME is set, then try to use it +### +if [[ -n "${YETUS_HOME}" + && -x "${YETUS_HOME}/bin/${WANTED}" ]]; then + exec "${YETUS_HOME}/bin/${WANTED}" "${ARGV[@]}" +fi + +# +# this directory is ignored by git and maven +# +HADOOP_PATCHPROCESS=${HADOOP_PATCHPROCESS:-"${BINDIR}/../../patchprocess"} + +if [[ ! -d "${HADOOP_PATCHPROCESS}" ]]; then + mkdir -p "${HADOOP_PATCHPROCESS}" +fi + +mytmpdir=$(yetus_abs "${HADOOP_PATCHPROCESS}") +if [[ $? != 0 ]]; then + yetus_error "yetus-dl: Unable to cwd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi +HADOOP_PATCHPROCESS=${mytmpdir} + +## +## if we've already DL'd it, then short cut +## +if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## need to DL, etc +## + +BASEURL="https://dist.apache.org/repos/dist/release/yetus/${HADOOP_YETUS_VERSION}/" +TARBALL="yetus-${HADOOP_YETUS_VERSION}-bin.tar" + +GPGBIN=$(command -v gpg) +CURLBIN=$(command -v curl) + +pushd "${HADOOP_PATCHPROCESS}" >/dev/null +if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi + +if [[ -n "${CURLBIN}" ]]; then + "${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz" + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: unable to download ${BASEURL}/${TARBALL}.gz" + exit 1 + fi +else + yetus_error "ERROR: yetus-dl requires curl." + exit 1 +fi + +if [[ -n "${GPGBIN}" ]]; then + mkdir -p .gpg + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: Unable to create ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + chmod -R 700 .gpg + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: Unable to chmod ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + "${CURLBIN}" -s -L -o KEYS_YETUS https://dist.apache.org/repos/dist/release/yetus/KEYS + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: unable to fetch https://dist.apache.org/repos/dist/release/yetus/KEYS" + exit 1 + fi + "${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz.asc" + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: unable to fetch ${BASEURL}/${TARBALL}.gz.asc" + exit 1 + fi + "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --import "${HADOOP_PATCHPROCESS}/KEYS_YETUS" >/dev/null 2>&1 + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: gpg unable to import ${HADOOP_PATCHPROCESS}/KEYS_YETUS" + exit 1 + fi + "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --verify "${TARBALL}.gz.asc" >/dev/null 2>&1 + if [[ $? != 0 ]]; then + yetus_error "ERROR: yetus-dl: gpg verify of tarball in ${HADOOP_PATCHPROCESS} failed" + exit 1 + fi +fi + +gunzip -c "${TARBALL}.gz" | tar xpf - +if [[ $? != 0 ]]; then + yetus_error "ERROR: ${TARBALL}.gz is corrupt. Investigate and then remove ${HADOOP_PATCHPROCESS} to try again." + exit 1 +fi + +if [[ -x "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + exec "${HADOOP_PATCHPROCESS}/yetus-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## give up +## +yetus_error "ERROR: ${WANTED} is not part of Apache Yetus ${HADOOP_YETUS_VERSION}" +exit 1