Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CD949200AC8 for ; Tue, 7 Jun 2016 18:59:54 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CC277160A36; Tue, 7 Jun 2016 16:59:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C9895160A4F for ; Tue, 7 Jun 2016 18:59:53 +0200 (CEST) Received: (qmail 29443 invoked by uid 500); 7 Jun 2016 16:59:53 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 29357 invoked by uid 99); 7 Jun 2016 16:59:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2016 16:59:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 9927418052A for ; Tue, 7 Jun 2016 16:59:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.396 X-Spam-Level: X-Spam-Status: No, score=-4.396 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id MRX6UrJOaSbw for ; Tue, 7 Jun 2016 16:59:48 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with SMTP id 41DC35F233 for ; Tue, 7 Jun 2016 16:59:47 +0000 (UTC) Received: (qmail 28704 invoked by uid 99); 7 Jun 2016 16:59:46 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2016 16:59:46 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 6B97BDFC7E; Tue, 7 Jun 2016 16:59:46 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: davor@apache.org To: commits@beam.incubator.apache.org Date: Tue, 07 Jun 2016 16:59:46 -0000 Message-Id: <584ff0d1e1df401a8ed4af046fe0db8e@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [01/11] incubator-beam git commit: Move 'travis/' directory into 'testing/travis/' archived-at: Tue, 07 Jun 2016 16:59:55 -0000 Repository: incubator-beam Updated Branches: refs/heads/release-0.1.0-incubating d6a3baa3b -> 54942431a Move 'travis/' directory into 'testing/travis/' Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eb529d45 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eb529d45 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/eb529d45 Branch: refs/heads/release-0.1.0-incubating Commit: eb529d45f86303fe4bcecd63a4abbde659a86856 Parents: fc009d6 Author: Davor Bonaci Authored: Thu Jun 2 15:09:53 2016 -0700 Committer: Davor Bonaci Committed: Mon Jun 6 18:08:21 2016 -0700 ---------------------------------------------------------------------- .travis.yml | 2 +- testing/travis/README.md | 23 +++++++ testing/travis/test_wordcount.sh | 125 ++++++++++++++++++++++++++++++++++ travis/README.md | 23 ------- travis/test_wordcount.sh | 125 ---------------------------------- 5 files changed, 149 insertions(+), 149 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb529d45/.travis.yml ---------------------------------------------------------------------- diff --git a/.travis.yml b/.travis.yml index ddf3204..6d81689 100644 --- a/.travis.yml +++ b/.travis.yml @@ -51,4 +51,4 @@ install: script: - travis_retry mvn -B $MAVEN_OVERRIDE install -U - - travis_retry travis/test_wordcount.sh + - travis_retry testing/travis/test_wordcount.sh http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb529d45/testing/travis/README.md ---------------------------------------------------------------------- diff --git a/testing/travis/README.md b/testing/travis/README.md new file mode 100644 index 0000000..e0c13f2 --- /dev/null +++ b/testing/travis/README.md @@ -0,0 +1,23 @@ + + +# Travis Scripts + +This directory contains scripts used for [Travis CI](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK) +testing. http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb529d45/testing/travis/test_wordcount.sh ---------------------------------------------------------------------- diff --git a/testing/travis/test_wordcount.sh b/testing/travis/test_wordcount.sh new file mode 100755 index 0000000..3efd37c --- /dev/null +++ b/testing/travis/test_wordcount.sh @@ -0,0 +1,125 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# This script runs WordCount example locally in a few different ways. +# Specifically, all combinations of: +# a) using mvn exec, or java -cp with a bundled jar file; +# b) input filename with no directory component, with a relative directory, or +# with an absolute directory; AND +# c) input filename containing wildcards or not. +# +# The one optional parameter is a path from the directory containing the script +# to the directory containing the top-level (parent) pom.xml. If no parameter +# is provided, the script assumes that directory is equal to the directory +# containing the script itself. +# +# The exit-code of the script indicates success or a failure. + +set -e +set -o pipefail + +PASS=1 +VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -v '\[') +JAR_FILE=examples/java/target/java-examples-all-bundled-${VERSION}.jar + +function check_result_hash { + local name=$1 + local outfile_prefix=$2 + local expected=$3 + + local actual=$(LC_ALL=C sort $outfile_prefix-* | md5sum | awk '{print $1}' \ + || LC_ALL=C sort $outfile_prefix-* | md5 -q) || exit 2 # OSX + if [[ "$actual" != "$expected" ]] + then + echo "FAIL $name: Output hash mismatch. Got $actual, expected $expected." + PASS="" + echo "head hexdump of actual:" + head $outfile_prefix-* | hexdump -c + else + echo "pass $name" + # Output files are left behind in /tmp + fi +} + +function get_outfile_prefix { + local name=$1 + # NOTE: mktemp on OSX doesn't support --tmpdir + mktemp -u "/tmp/$name.out.XXXXXXXXXX" +} + +function run_via_mvn { + local name=$1 + local input=$2 + local expected_hash=$3 + + local outfile_prefix="$(get_outfile_prefix "$name")" || exit 2 + local cmd='mvn exec:java -f pom.xml -pl examples/java \ + -Dexec.mainClass=org.apache.beam.examples.WordCount \ + -Dexec.args="--runner=DirectPipelineRunner --inputFile='"$input"' --output='"$outfile_prefix"'"' + echo "$name: Running $cmd" >&2 + sh -c "$cmd" + check_result_hash "$name" "$outfile_prefix" "$expected_hash" +} + +function run_bundled { + local name=$1 + local input=$2 + local expected_hash=$3 + + local outfile_prefix="$(get_outfile_prefix "$name")" || exit 2 + local cmd='java -cp '"$JAR_FILE"' \ + org.apache.beam.examples.WordCount \ + --runner=DirectPipelineRunner \ + --inputFile='"'$input'"' \ + --output='"$outfile_prefix" + echo "$name: Running $cmd" >&2 + sh -c "$cmd" + check_result_hash "$name" "$outfile_prefix" "$expected_hash" +} + +function run_all_ways { + local name=$1 + local input=$2 + local expected_hash=$3 + + run_via_mvn ${name}a "$input" $expected_hash + check_for_jar_file + run_bundled ${name}b "$input" $expected_hash +} + +function check_for_jar_file { + if [[ ! -f $JAR_FILE ]] + then + echo "Jar file $JAR_FILE not created" >&2 + exit 2 + fi +} + +run_all_ways wordcount1 "LICENSE" c5350a5ad4bb51e3e018612b4b044097 +run_all_ways wordcount2 "./LICENSE" c5350a5ad4bb51e3e018612b4b044097 +run_all_ways wordcount3 "$PWD/LICENSE" c5350a5ad4bb51e3e018612b4b044097 +run_all_ways wordcount4 "L*N?E*" c5350a5ad4bb51e3e018612b4b044097 +run_all_ways wordcount5 "./LICE*N?E" c5350a5ad4bb51e3e018612b4b044097 +run_all_ways wordcount6 "$PWD/*LIC?NSE" c5350a5ad4bb51e3e018612b4b044097 + +if [[ ! "$PASS" ]] +then + echo "One or more tests FAILED." + exit 1 +fi +echo "All tests PASS" http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb529d45/travis/README.md ---------------------------------------------------------------------- diff --git a/travis/README.md b/travis/README.md deleted file mode 100644 index e0c13f2..0000000 --- a/travis/README.md +++ /dev/null @@ -1,23 +0,0 @@ - - -# Travis Scripts - -This directory contains scripts used for [Travis CI](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK) -testing. http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eb529d45/travis/test_wordcount.sh ---------------------------------------------------------------------- diff --git a/travis/test_wordcount.sh b/travis/test_wordcount.sh deleted file mode 100755 index 3efd37c..0000000 --- a/travis/test_wordcount.sh +++ /dev/null @@ -1,125 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# This script runs WordCount example locally in a few different ways. -# Specifically, all combinations of: -# a) using mvn exec, or java -cp with a bundled jar file; -# b) input filename with no directory component, with a relative directory, or -# with an absolute directory; AND -# c) input filename containing wildcards or not. -# -# The one optional parameter is a path from the directory containing the script -# to the directory containing the top-level (parent) pom.xml. If no parameter -# is provided, the script assumes that directory is equal to the directory -# containing the script itself. -# -# The exit-code of the script indicates success or a failure. - -set -e -set -o pipefail - -PASS=1 -VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -v '\[') -JAR_FILE=examples/java/target/java-examples-all-bundled-${VERSION}.jar - -function check_result_hash { - local name=$1 - local outfile_prefix=$2 - local expected=$3 - - local actual=$(LC_ALL=C sort $outfile_prefix-* | md5sum | awk '{print $1}' \ - || LC_ALL=C sort $outfile_prefix-* | md5 -q) || exit 2 # OSX - if [[ "$actual" != "$expected" ]] - then - echo "FAIL $name: Output hash mismatch. Got $actual, expected $expected." - PASS="" - echo "head hexdump of actual:" - head $outfile_prefix-* | hexdump -c - else - echo "pass $name" - # Output files are left behind in /tmp - fi -} - -function get_outfile_prefix { - local name=$1 - # NOTE: mktemp on OSX doesn't support --tmpdir - mktemp -u "/tmp/$name.out.XXXXXXXXXX" -} - -function run_via_mvn { - local name=$1 - local input=$2 - local expected_hash=$3 - - local outfile_prefix="$(get_outfile_prefix "$name")" || exit 2 - local cmd='mvn exec:java -f pom.xml -pl examples/java \ - -Dexec.mainClass=org.apache.beam.examples.WordCount \ - -Dexec.args="--runner=DirectPipelineRunner --inputFile='"$input"' --output='"$outfile_prefix"'"' - echo "$name: Running $cmd" >&2 - sh -c "$cmd" - check_result_hash "$name" "$outfile_prefix" "$expected_hash" -} - -function run_bundled { - local name=$1 - local input=$2 - local expected_hash=$3 - - local outfile_prefix="$(get_outfile_prefix "$name")" || exit 2 - local cmd='java -cp '"$JAR_FILE"' \ - org.apache.beam.examples.WordCount \ - --runner=DirectPipelineRunner \ - --inputFile='"'$input'"' \ - --output='"$outfile_prefix" - echo "$name: Running $cmd" >&2 - sh -c "$cmd" - check_result_hash "$name" "$outfile_prefix" "$expected_hash" -} - -function run_all_ways { - local name=$1 - local input=$2 - local expected_hash=$3 - - run_via_mvn ${name}a "$input" $expected_hash - check_for_jar_file - run_bundled ${name}b "$input" $expected_hash -} - -function check_for_jar_file { - if [[ ! -f $JAR_FILE ]] - then - echo "Jar file $JAR_FILE not created" >&2 - exit 2 - fi -} - -run_all_ways wordcount1 "LICENSE" c5350a5ad4bb51e3e018612b4b044097 -run_all_ways wordcount2 "./LICENSE" c5350a5ad4bb51e3e018612b4b044097 -run_all_ways wordcount3 "$PWD/LICENSE" c5350a5ad4bb51e3e018612b4b044097 -run_all_ways wordcount4 "L*N?E*" c5350a5ad4bb51e3e018612b4b044097 -run_all_ways wordcount5 "./LICE*N?E" c5350a5ad4bb51e3e018612b4b044097 -run_all_ways wordcount6 "$PWD/*LIC?NSE" c5350a5ad4bb51e3e018612b4b044097 - -if [[ ! "$PASS" ]] -then - echo "One or more tests FAILED." - exit 1 -fi -echo "All tests PASS"