Return-Path: X-Original-To: apmail-bigtop-commits-archive@www.apache.org Delivered-To: apmail-bigtop-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3EB2F18E74 for ; Tue, 12 Jan 2016 23:47:15 +0000 (UTC) Received: (qmail 79877 invoked by uid 500); 12 Jan 2016 23:47:15 -0000 Delivered-To: apmail-bigtop-commits-archive@bigtop.apache.org Received: (qmail 79822 invoked by uid 500); 12 Jan 2016 23:47:15 -0000 Mailing-List: contact commits-help@bigtop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: bigtop-dev@bigtop.apache.org Delivered-To: mailing list commits@bigtop.apache.org Received: (qmail 79813 invoked by uid 99); 12 Jan 2016 23:47:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2016 23:47:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id C3A87E2C5B; Tue, 12 Jan 2016 23:47:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: cos@apache.org To: commits@bigtop.apache.org Message-Id: <0c65dccde331478bacbaf32cfe10d2a5@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: bigtop git commit: BIGTOP-2227. smoke-tests should be a part of the top-level project, not separated Date: Tue, 12 Jan 2016 23:47:14 +0000 (UTC) Repository: bigtop Updated Branches: refs/heads/master 6c9829e8b -> 5d63481d5 BIGTOP-2227. smoke-tests should be a part of the top-level project, not separated Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/5d63481d Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/5d63481d Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/5d63481d Branch: refs/heads/master Commit: 5d63481d533e79f89f2be56dede58c7c2bb29fc5 Parents: 6c9829e Author: Konstantin Boudnik Authored: Fri Jan 8 21:29:46 2016 -0800 Committer: Konstantin Boudnik Committed: Tue Jan 12 15:44:20 2016 -0800 ---------------------------------------------------------------------- README.md | 5 +- bigtop-deploy/vm/utils/smoke-tests.sh | 8 ++-- bigtop-tests/smoke-tests/README | 25 +++++++--- bigtop-tests/smoke-tests/build.gradle | 22 +++------ bigtop-tests/smoke-tests/hive/build.gradle | 8 ++-- .../smoke-tests/ignite-hadoop/build.gradle | 4 +- bigtop-tests/smoke-tests/mahout/build.gradle | 4 +- bigtop-tests/smoke-tests/mapreduce/build.gradle | 2 +- bigtop-tests/smoke-tests/pig/build.gradle | 2 +- bigtop-tests/smoke-tests/settings.gradle | 50 -------------------- build.gradle | 21 +++++++- settings.gradle | 14 +++++- 12 files changed, 73 insertions(+), 92 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 8c08616..e772809 100644 --- a/README.md +++ b/README.md @@ -97,10 +97,7 @@ For specific questions it's always a good idea to ping the mailing list at dev-s For Users: Running the smoke tests. ----------------------------------- -The simplest way to test bigtop is to: - -* Step 1: cd bigtop-tests/smoke-tests/ -* Step 2: Follow the instructions in the smoke-tests/README file. +The simplest way to test bigtop is described in bigtop-tests/smoke-tests/README file For integration (API level) testing with maven, read on. http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-deploy/vm/utils/smoke-tests.sh ---------------------------------------------------------------------- diff --git a/bigtop-deploy/vm/utils/smoke-tests.sh b/bigtop-deploy/vm/utils/smoke-tests.sh index b27c082..1d83687 100755 --- a/bigtop-deploy/vm/utils/smoke-tests.sh +++ b/bigtop-deploy/vm/utils/smoke-tests.sh @@ -28,7 +28,6 @@ fi echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n" export HADOOP_CONF_DIR=/etc/hadoop/conf/ -export BIGTOP_HOME=/bigtop-home/ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/ export HIVE_HOME=/usr/lib/hive/ export PIG_HOME=/usr/lib/pig/ @@ -36,7 +35,6 @@ export FLUME_HOME=/usr/lib/flume/ export SQOOP_HOME=/usr/lib/sqoop/ export HIVE_CONF_DIR=/etc/hive/conf/ export MAHOUT_HOME="/usr/lib/mahout" -export ITEST="1.0.0" su -s /bin/bash $HCFS_USER -c '/usr/bin/hadoop fs -mkdir /user/vagrant /user/root' su -s /bin/bash $HCFS_USER -c 'hadoop fs -chmod 777 /user/vagrant' @@ -47,4 +45,8 @@ if [ -f /etc/debian_version ] ; then else yum install -y pig hive flume mahout sqoop fi -cd /bigtop-home && ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=$SMOKE_TESTS --info +ALL_SMOKE_TASKS="" +for s in `echo $SMOKE_TESTS | sed -e 's#,# #g'`; do + ALL_SMOKE_TASKS="$ALL_SMOKE_TASKS bigtop-tests:smoke-tests:$s:test" +done +cd /bigtop-home && ./gradlew clean $ALL_SMOKE_TASKS -Psmoke.tests --info http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/README ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/README b/bigtop-tests/smoke-tests/README index 992f49c..430d03b 100644 --- a/bigtop-tests/smoke-tests/README +++ b/bigtop-tests/smoke-tests/README @@ -18,7 +18,6 @@ So overall, this framework makes testing with apache bigtop easy as well as fun! You need to export several environment variables. For example, ``` export HADOOP_CONF_DIR=/etc/hadoop/conf/ -export BIGTOP_HOME=/bigtop-home/ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/ export HIVE_HOME=/usr/lib/hive/ export PIG_HOME=/usr/lib/pig/ @@ -28,19 +27,33 @@ export JAVA_HOME="/usr/lib/jvm/java-openjdk/" export MAHOUT_HOME="/usr/lib/mahout" export KITE_HOME="/usr/lib/kite" export SPARK_HOME="/usr/lib/spark" -export ITEST="1.0.0" ``` -Then, simply invoke the tests including the smoke.tests you want to run, as described below. +Then, simply invoke the tests you want to run, as described below. Each directory is a gradle "subproject" mapping to an ecosystem component. -If you specify, for example, "flume", then the flume test runs. To test hive as well, -you would specify "flume,hive", and so on. To run the tests from the top-level +Invoking test task of flume subproject will execute flume tests. To test hive as well, +you would add the invocation of hive:test, and so on. To run the tests from the top-level Bigtop folder, follow the example below: ``` - ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=flume,hive --info + ./gradlew bigtop-tests:smoke-tests:flume:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests --info ``` +You also can run smoke tests from bigtop-tests/smoke-tests directory, +or from any subproject of it. E.g. +``` + cd bigtop-tests/smoke-tests + ../../gradlew hive:test -Psmoke.tests +``` +or +``` + cd bigtop-tests/smoke-tests/hive + ../../../gradlew test -Psmoke.tests +``` +Instead of using top-level Gradle wrapper one can also use system-wide +Gradle, if installed and available. +Setting up project property smoke.tests is important, cause this is +required to to enable test tasks for smoke projects. The --info option is a nice feature to have here: You can easily debug your tests this way, just make sure you have log4j.properties in your conf/ directory locally. http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/build.gradle b/bigtop-tests/smoke-tests/build.gradle index 1e0da58..6842c51 100644 --- a/bigtop-tests/smoke-tests/build.gradle +++ b/bigtop-tests/smoke-tests/build.gradle @@ -15,14 +15,6 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -allprojects { - apply plugin: 'groovy' - - repositories { - mavenCentral() - } -} - subprojects { /** * Utility function for tests to use to confirm EVN Variables. @@ -32,21 +24,20 @@ subprojects { def value = System.getenv("${it}") if (value == null || value == "null") throw new GradleException("undeclared env variable: ${it}") - else - println("ENV VARIABLE: ${it} = $value"); } } /** * Two important environment variables. * java_home should be declared in all cases. - * itest can be greater than or equal to = 0.7.0, 0.8.0 */ - checkEnv(["JAVA_HOME","ITEST","BIGTOP_HOME"]) + checkEnv(["JAVA_HOME"]) - ext.itestVersion = System.getenv("ITEST") ext.groovyVersion = '1.8.0' ext.hadoopVersion = '2.6.0' + // itest needs be greater than or equal to = 1.0.0 + ext.itestVersion = '1.0.0' // Might need to be able to read an input for alternate version? + ext.BIGTOP_HOME = rootDir dependencies { //needed to avoid groovy not on classpath error. @@ -81,9 +72,8 @@ subprojects { return !keep_this_test; } - println("Now testing..."); - test { - + test << { + println("Now testing..."); //todo, add back in 'basic' after BIGTOP-1392 . testLogging { events "passed", "skipped", "failed" http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/hive/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/hive/build.gradle b/bigtop-tests/smoke-tests/hive/build.gradle index 67dcb71..9b06c3c 100644 --- a/bigtop-tests/smoke-tests/hive/build.gradle +++ b/bigtop-tests/smoke-tests/hive/build.gradle @@ -26,8 +26,8 @@ def tests_to_include() { sourceSets { test { groovy { - srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/", - "${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"] + srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/", + "${BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"] exclude { FileTreeElement elem -> (doExclude(elem.getName())) } } } @@ -39,7 +39,5 @@ test { } test.doFirst { - // BIGTOP_HOME is necessary here, since we use it to read in from - // test-artifacts. - checkEnv(["BIGTOP_HOME", "HIVE_HOME", "HIVE_CONF_DIR", "BIGTOP_HOME"]) + checkEnv(["HIVE_HOME", "HIVE_CONF_DIR"]) } http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle index b3ea841..46e6364 100644 --- a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle +++ b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle @@ -24,11 +24,11 @@ def tests_to_include() { sourceSets { test { groovy { - srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"] + srcDirs = ["${BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"] } } } test.doFirst { - checkEnv(["BIGTOP_HOME", "HADOOP_MAPRED_HOME"]) + checkEnv(["HADOOP_MAPRED_HOME"]) } http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mahout/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/mahout/build.gradle b/bigtop-tests/smoke-tests/mahout/build.gradle index b6048d6..d79c09e 100644 --- a/bigtop-tests/smoke-tests/mahout/build.gradle +++ b/bigtop-tests/smoke-tests/mahout/build.gradle @@ -24,12 +24,12 @@ def tests_to_include() { sourceSets { test { groovy { - srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"] + srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"] exclude { FileTreeElement elem -> (doExclude(elem.getName())) } } } } test.doFirst { - checkEnv(["BIGTOP_HOME", "MAHOUT_HOME"]) + checkEnv(["MAHOUT_HOME"]) } http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mapreduce/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/mapreduce/build.gradle b/bigtop-tests/smoke-tests/mapreduce/build.gradle index 4aad5f4..0d5e4df 100644 --- a/bigtop-tests/smoke-tests/mapreduce/build.gradle +++ b/bigtop-tests/smoke-tests/mapreduce/build.gradle @@ -25,7 +25,7 @@ sourceSets { test { groovy { srcDirs = [ - "${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce" + "${BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce" ] exclude { FileTreeElement elem -> (doExclude(elem.getName())) http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/pig/build.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/pig/build.gradle b/bigtop-tests/smoke-tests/pig/build.gradle index 81c0d68..cc09ea8 100644 --- a/bigtop-tests/smoke-tests/pig/build.gradle +++ b/bigtop-tests/smoke-tests/pig/build.gradle @@ -26,7 +26,7 @@ def tests_to_include() { sourceSets { test { groovy { - srcDirs = ["./"] + srcDirs = [projectDir] exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**' exclude { FileTreeElement elem -> (doExclude(elem.getName())) } } http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/settings.gradle ---------------------------------------------------------------------- diff --git a/bigtop-tests/smoke-tests/settings.gradle b/bigtop-tests/smoke-tests/settings.gradle deleted file mode 100644 index 464988c..0000000 --- a/bigtop-tests/smoke-tests/settings.gradle +++ /dev/null @@ -1,50 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - *

- * http://www.apache.org/licenses/LICENSE-2.0 - *

- * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -def tests_comma_str = System.properties['smoke.tests'] - -if(tests_comma_str != null) { - def tests = tests_comma_str.split(",") - tests.each() { t -> - println("including $t") - include(t) - } -} -else { - println(""" - - ****************************************************************************** - Welcome to the bigtop smoke tests. - - This is an extensible test module, with one directory for every test component. - - To run it, just use the following command, adding components (directories) into - - the smoke.tests system property. - - For example, to run the hive/ and pig/ smoke tests: - - gradle clean compileGroovy test -Dsmoke.tests=hive,pig - - NOTE: You must run clean. Otherwise gradle might skip some of the tests. - - ... Exiting now ... - ****************************************************************************** - """) - System.exit(1) -} http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/build.gradle ---------------------------------------------------------------------- diff --git a/build.gradle b/build.gradle index 56accb9..e7cd663 100644 --- a/build.gradle +++ b/build.gradle @@ -172,6 +172,24 @@ project(':itest-common') { } } +/** + * checkSmokeTestProjects() configures smoke-tests::test task only + * in if the project has smoke.tests properties set. + * This is done to avoid running cluster smoke tests during the normal life-cycle + * of the project development + */ +def checkSmokeTestProjects = { + FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle') + fTree.each() { smokeProject -> + def parent = smokeProject.getParentFile().name + project (":bigtop-tests:smoke-tests:$parent") { + test { + onlyIf {project.hasProperty('smoke.tests')} + } + } + } +} + def TESTARTIFACTS_GROUP = 'test artifacts' def DEVENV_GROUP = 'development tools' def DEPLOY_GROUP = 'deployment' @@ -389,7 +407,8 @@ task "bigtop-slaves"(dependsOn: 'gen-gradle-home', type:Exec, commandLine command } -project.afterEvaluate{ +project.afterEvaluate { + checkSmokeTestProjects() artifactToInstall(dependsOn: [installTopLevel, installCommon, installConf, installiTest]) } http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/settings.gradle ---------------------------------------------------------------------- diff --git a/settings.gradle b/settings.gradle index 30e6924..ac071f1 100644 --- a/settings.gradle +++ b/settings.gradle @@ -18,5 +18,17 @@ rootProject.name = 'bigtop' include 'bigtop-test-framework', 'bigtop-tests:smoke-tests' - project(":bigtop-test-framework").name = 'itest-common' + +/** + * The following code will does include all subprojects of smoke-tests + * so they are lined-up properly in the structure of the Bigtop project. + * The inclusion is done dynamically, so when new smoke test projects + * there's no extra effort to get it running. + */ +FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle') +fTree.each() { buildFile -> + def parent = buildFile.getParentFile().name + include("bigtop-tests:smoke-tests:$parent") +} +