spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-9866][SQL] Speed up VersionsSuite by using persistent Ivy cache
Date Tue, 24 Nov 2015 00:33:32 GMT
Repository: spark
Updated Branches:
  refs/heads/master c2467dada -> 9db5f601f


[SPARK-9866][SQL] Speed up VersionsSuite by using persistent Ivy cache

This patch attempts to speed up VersionsSuite by storing fetched Hive JARs in an Ivy cache
that persists across tests runs. If `SPARK_VERSIONS_SUITE_IVY_PATH` is set, that path will
be used for the cache; if it is not set, VersionsSuite will create a temporary Ivy cache which
is deleted after the test completes.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #9624 from JoshRosen/SPARK-9866.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9db5f601
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9db5f601
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9db5f601

Branch: refs/heads/master
Commit: 9db5f601facfdaba6e4333a6b2d2e4a9f009c788
Parents: c2467da
Author: Josh Rosen <joshrosen@databricks.com>
Authored: Mon Nov 23 16:33:26 2015 -0800
Committer: Josh Rosen <joshrosen@databricks.com>
Committed: Mon Nov 23 16:33:26 2015 -0800

----------------------------------------------------------------------
 .../org/apache/spark/sql/hive/client/VersionsSuite.scala  | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/9db5f601/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
index c6d034a..7bc13bc 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
@@ -36,10 +36,12 @@ import org.apache.spark.util.Utils
 @ExtendedHiveTest
 class VersionsSuite extends SparkFunSuite with Logging {
 
-  // Do not use a temp path here to speed up subsequent executions of the unit test during
-  // development.
-  private val ivyPath = Some(
-    new File(sys.props("java.io.tmpdir"), "hive-ivy-cache").getAbsolutePath())
+  // In order to speed up test execution during development or in Jenkins, you can specify
the path
+  // of an existing Ivy cache:
+  private val ivyPath: Option[String] = {
+    sys.env.get("SPARK_VERSIONS_SUITE_IVY_PATH").orElse(
+      Some(new File(sys.props("java.io.tmpdir"), "hive-ivy-cache").getAbsolutePath))
+  }
 
   private def buildConf() = {
     lazy val warehousePath = Utils.createTempDir()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message