bigtop-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From c..@apache.org
Subject bigtop git commit: BIGTOP-2227. smoke-tests should be a part of the top-level project, not separated
Date Tue, 12 Jan 2016 23:47:14 GMT
Repository: bigtop
Updated Branches:
  refs/heads/master 6c9829e8b -> 5d63481d5


BIGTOP-2227. smoke-tests should be a part of the top-level project, not separated


Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/5d63481d
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/5d63481d
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/5d63481d

Branch: refs/heads/master
Commit: 5d63481d533e79f89f2be56dede58c7c2bb29fc5
Parents: 6c9829e
Author: Konstantin Boudnik <cos@apache.org>
Authored: Fri Jan 8 21:29:46 2016 -0800
Committer: Konstantin Boudnik <cos@apache.org>
Committed: Tue Jan 12 15:44:20 2016 -0800

----------------------------------------------------------------------
 README.md                                       |  5 +-
 bigtop-deploy/vm/utils/smoke-tests.sh           |  8 ++--
 bigtop-tests/smoke-tests/README                 | 25 +++++++---
 bigtop-tests/smoke-tests/build.gradle           | 22 +++------
 bigtop-tests/smoke-tests/hive/build.gradle      |  8 ++--
 .../smoke-tests/ignite-hadoop/build.gradle      |  4 +-
 bigtop-tests/smoke-tests/mahout/build.gradle    |  4 +-
 bigtop-tests/smoke-tests/mapreduce/build.gradle |  2 +-
 bigtop-tests/smoke-tests/pig/build.gradle       |  2 +-
 bigtop-tests/smoke-tests/settings.gradle        | 50 --------------------
 build.gradle                                    | 21 +++++++-
 settings.gradle                                 | 14 +++++-
 12 files changed, 73 insertions(+), 92 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 8c08616..e772809 100644
--- a/README.md
+++ b/README.md
@@ -97,10 +97,7 @@ For specific questions it's always a good idea to ping the mailing list
at dev-s
 For Users: Running the smoke tests.
 -----------------------------------
 
-The simplest way to test bigtop is to:
-
-* Step 1: cd bigtop-tests/smoke-tests/
-* Step 2: Follow the instructions in the smoke-tests/README file.
+The simplest way to test bigtop is described in bigtop-tests/smoke-tests/README file
 
 For integration (API level) testing with maven, read on. 
 

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-deploy/vm/utils/smoke-tests.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/utils/smoke-tests.sh b/bigtop-deploy/vm/utils/smoke-tests.sh
index b27c082..1d83687 100755
--- a/bigtop-deploy/vm/utils/smoke-tests.sh
+++ b/bigtop-deploy/vm/utils/smoke-tests.sh
@@ -28,7 +28,6 @@ fi
 echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"
 
 export HADOOP_CONF_DIR=/etc/hadoop/conf/
-export BIGTOP_HOME=/bigtop-home/
 export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
 export HIVE_HOME=/usr/lib/hive/
 export PIG_HOME=/usr/lib/pig/
@@ -36,7 +35,6 @@ export FLUME_HOME=/usr/lib/flume/
 export SQOOP_HOME=/usr/lib/sqoop/
 export HIVE_CONF_DIR=/etc/hive/conf/
 export MAHOUT_HOME="/usr/lib/mahout"
-export ITEST="1.0.0"
 
 su -s /bin/bash $HCFS_USER -c '/usr/bin/hadoop fs -mkdir /user/vagrant /user/root'
 su -s /bin/bash $HCFS_USER -c 'hadoop fs -chmod 777 /user/vagrant'
@@ -47,4 +45,8 @@ if [ -f /etc/debian_version ] ; then
 else
     yum install -y pig hive flume mahout sqoop
 fi
-cd /bigtop-home && ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test
-Dsmoke.tests=$SMOKE_TESTS --info
+ALL_SMOKE_TASKS=""
+for s in `echo $SMOKE_TESTS | sed -e 's#,# #g'`; do
+  ALL_SMOKE_TASKS="$ALL_SMOKE_TASKS bigtop-tests:smoke-tests:$s:test"
+done
+cd /bigtop-home && ./gradlew clean $ALL_SMOKE_TASKS -Psmoke.tests --info

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/README
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/README b/bigtop-tests/smoke-tests/README
index 992f49c..430d03b 100644
--- a/bigtop-tests/smoke-tests/README
+++ b/bigtop-tests/smoke-tests/README
@@ -18,7 +18,6 @@ So overall, this framework makes testing with apache bigtop easy as well
as fun!
 You need to export several environment variables.   For example,
 ```
 export HADOOP_CONF_DIR=/etc/hadoop/conf/
-export BIGTOP_HOME=/bigtop-home/
 export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
 export HIVE_HOME=/usr/lib/hive/
 export PIG_HOME=/usr/lib/pig/
@@ -28,19 +27,33 @@ export JAVA_HOME="/usr/lib/jvm/java-openjdk/"
 export MAHOUT_HOME="/usr/lib/mahout"
 export KITE_HOME="/usr/lib/kite"
 export SPARK_HOME="/usr/lib/spark"
-export ITEST="1.0.0"
 ```
 
-Then, simply invoke the tests including the smoke.tests you want to run, as described below.
+Then, simply invoke the tests you want to run, as described below.
 
 Each directory is a gradle "subproject" mapping to an ecosystem component.
-If you specify, for example, "flume", then the flume test runs.  To test hive as well,
-you would specify "flume,hive", and so on. To run the tests from the top-level
+Invoking test task of flume subproject will execute flume tests. To test hive as well,
+you would add the invocation of hive:test, and so on. To run the tests from the top-level
 Bigtop folder, follow the example below:
 ```
-    ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=flume,hive
--info
+    ./gradlew bigtop-tests:smoke-tests:flume:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests
--info
 ```
+You also can run smoke tests from bigtop-tests/smoke-tests directory,
+or from any subproject of it. E.g.
+```
+    cd bigtop-tests/smoke-tests
+    ../../gradlew hive:test -Psmoke.tests
+```
+or
+```
+    cd bigtop-tests/smoke-tests/hive
+    ../../../gradlew test -Psmoke.tests
+```
+Instead of using top-level Gradle wrapper one can also use system-wide
+Gradle, if installed and available.
 
+Setting up project property smoke.tests is important, cause this is
+required to to enable test tasks for smoke projects.
 The --info option is a nice feature to have here:  You can easily debug your tests this way,
 just make sure you have log4j.properties in your conf/ directory locally.
 

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/build.gradle b/bigtop-tests/smoke-tests/build.gradle
index 1e0da58..6842c51 100644
--- a/bigtop-tests/smoke-tests/build.gradle
+++ b/bigtop-tests/smoke-tests/build.gradle
@@ -15,14 +15,6 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-allprojects {
-  apply plugin: 'groovy'
-
-  repositories {
-    mavenCentral()
-  }
-}
-
 subprojects {
   /**
    *  Utility function for tests to use to confirm EVN Variables.
@@ -32,21 +24,20 @@ subprojects {
       def value = System.getenv("${it}")
       if (value == null || value == "null")
         throw new GradleException("undeclared env variable: ${it}")
-      else
-        println("ENV VARIABLE: ${it} = $value");
     }
   }
 
   /**
   * Two important environment variables.
   *  java_home should be declared in all cases.
-  * itest can be greater than or equal to = 0.7.0, 0.8.0
   */
-  checkEnv(["JAVA_HOME","ITEST","BIGTOP_HOME"])
+  checkEnv(["JAVA_HOME"])
 
-  ext.itestVersion = System.getenv("ITEST")
   ext.groovyVersion = '1.8.0'
   ext.hadoopVersion = '2.6.0'
+  // itest needs be greater than or equal to = 1.0.0
+  ext.itestVersion = '1.0.0' // Might need to be able to read an input for alternate version?
+  ext.BIGTOP_HOME   = rootDir
 
   dependencies {
     //needed to avoid groovy not on classpath error.
@@ -81,9 +72,8 @@ subprojects {
     return !keep_this_test;
   }
 
-  println("Now testing...");
-  test {
-
+  test << {
+    println("Now testing...");
     //todo, add back in 'basic' after BIGTOP-1392 .
     testLogging {
       events "passed", "skipped", "failed"

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/hive/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/hive/build.gradle b/bigtop-tests/smoke-tests/hive/build.gradle
index 67dcb71..9b06c3c 100644
--- a/bigtop-tests/smoke-tests/hive/build.gradle
+++ b/bigtop-tests/smoke-tests/hive/build.gradle
@@ -26,8 +26,8 @@ def tests_to_include() {
 sourceSets {
   test {
     groovy {
-      srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
-                 "${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
+      srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
+                 "${BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
       exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
     }
   }
@@ -39,7 +39,5 @@ test {
 }
 
 test.doFirst {
-  // BIGTOP_HOME is necessary here, since we use it to read in from
-  // test-artifacts.
-  checkEnv(["BIGTOP_HOME", "HIVE_HOME", "HIVE_CONF_DIR", "BIGTOP_HOME"])
+  checkEnv(["HIVE_HOME", "HIVE_CONF_DIR"])
 }

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
index b3ea841..46e6364 100644
--- a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
+++ b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
@@ -24,11 +24,11 @@ def tests_to_include() {
 sourceSets {
   test {
     groovy {
-        srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
+        srcDirs = ["${BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
     }
   }
 }
 
 test.doFirst {
-  checkEnv(["BIGTOP_HOME", "HADOOP_MAPRED_HOME"])
+  checkEnv(["HADOOP_MAPRED_HOME"])
 }

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mahout/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/mahout/build.gradle b/bigtop-tests/smoke-tests/mahout/build.gradle
index b6048d6..d79c09e 100644
--- a/bigtop-tests/smoke-tests/mahout/build.gradle
+++ b/bigtop-tests/smoke-tests/mahout/build.gradle
@@ -24,12 +24,12 @@ def tests_to_include() {
 sourceSets {
   test {
     groovy {
-      srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
+      srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
       exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
     }
   }
 }
 
 test.doFirst {
-  checkEnv(["BIGTOP_HOME", "MAHOUT_HOME"])
+  checkEnv(["MAHOUT_HOME"])
 }

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mapreduce/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/mapreduce/build.gradle b/bigtop-tests/smoke-tests/mapreduce/build.gradle
index 4aad5f4..0d5e4df 100644
--- a/bigtop-tests/smoke-tests/mapreduce/build.gradle
+++ b/bigtop-tests/smoke-tests/mapreduce/build.gradle
@@ -25,7 +25,7 @@ sourceSets {
   test {
     groovy {
       srcDirs = [
-        "${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
+        "${BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
       ]
       exclude {
         FileTreeElement elem -> (doExclude(elem.getName()))

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/pig/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/pig/build.gradle b/bigtop-tests/smoke-tests/pig/build.gradle
index 81c0d68..cc09ea8 100644
--- a/bigtop-tests/smoke-tests/pig/build.gradle
+++ b/bigtop-tests/smoke-tests/pig/build.gradle
@@ -26,7 +26,7 @@ def tests_to_include() {
 sourceSets {
   test {
     groovy {
-      srcDirs = ["./"]
+      srcDirs = [projectDir]
       exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
       exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
     }

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/settings.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/settings.gradle b/bigtop-tests/smoke-tests/settings.gradle
deleted file mode 100644
index 464988c..0000000
--- a/bigtop-tests/smoke-tests/settings.gradle
+++ /dev/null
@@ -1,50 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * <p/>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p/>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def tests_comma_str = System.properties['smoke.tests']
-
-if(tests_comma_str != null) {
-  def tests = tests_comma_str.split(",")
-  tests.each() { t ->
-    println("including $t")
-    include(t)
-  }
-}
-else {
-  println("""
-
-     ******************************************************************************
-     Welcome to the bigtop smoke tests.
-
-     This is an extensible test module, with one directory for every test component.
-
-     To run it, just use the following command, adding components (directories) into
-
-     the smoke.tests system property.
-
-     For example, to run the hive/ and pig/ smoke tests:
-
-         gradle clean compileGroovy test -Dsmoke.tests=hive,pig
-
-     NOTE:  You must run clean.  Otherwise gradle might skip some of the tests.
-
-     ... Exiting now ...
-     ******************************************************************************
-  """)
-  System.exit(1)
-}

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index 56accb9..e7cd663 100644
--- a/build.gradle
+++ b/build.gradle
@@ -172,6 +172,24 @@ project(':itest-common') {
   }
 }
 
+/**
+ * checkSmokeTestProjects() configures smoke-tests:<component>:test task only
+ * in if the project has smoke.tests properties set.
+ * This is done to avoid running cluster smoke tests during the normal life-cycle
+ * of the project development
+ */
+def checkSmokeTestProjects = {
+  FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
+  fTree.each() { smokeProject ->
+    def parent = smokeProject.getParentFile().name
+    project (":bigtop-tests:smoke-tests:$parent") {
+      test {
+        onlyIf {project.hasProperty('smoke.tests')}
+      }
+    }
+  }
+}
+
 def TESTARTIFACTS_GROUP = 'test artifacts'
 def DEVENV_GROUP = 'development tools'
 def DEPLOY_GROUP = 'deployment'
@@ -389,7 +407,8 @@ task "bigtop-slaves"(dependsOn: 'gen-gradle-home', type:Exec,
   commandLine command
 }
 
-project.afterEvaluate{
+project.afterEvaluate {
+  checkSmokeTestProjects()
   artifactToInstall(dependsOn: [installTopLevel, installCommon, installConf, installiTest])
 }
 

http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/settings.gradle
----------------------------------------------------------------------
diff --git a/settings.gradle b/settings.gradle
index 30e6924..ac071f1 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -18,5 +18,17 @@
 rootProject.name = 'bigtop'
 
 include 'bigtop-test-framework', 'bigtop-tests:smoke-tests'
-
 project(":bigtop-test-framework").name = 'itest-common'
+
+/**
+ * The following code will does include all subprojects of smoke-tests
+ * so they are lined-up properly in the structure of the Bigtop project.
+ * The inclusion is done dynamically, so when new smoke test projects
+ * there's no extra effort to get it running.
+ */
+FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
+fTree.each() { buildFile ->
+  def parent = buildFile.getParentFile().name
+  include("bigtop-tests:smoke-tests:$parent")
+}
+


Mime
View raw message