spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-21936][SQL][FOLLOW-UP] backward compatibility test framework for HiveExternalCatalog
Date Tue, 24 Oct 2017 12:56:40 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 bf8163f5b -> daa838b88


[SPARK-21936][SQL][FOLLOW-UP] backward compatibility test framework for HiveExternalCatalog

## What changes were proposed in this pull request?

Adjust Spark download in test to use Apache mirrors and respect its load balancer, and use
Spark 2.1.2. This follows on a recent PMC list thread about removing the cloudfront download
rather than update it further.

## How was this patch tested?

Existing tests.

Author: Sean Owen <sowen@cloudera.com>

Closes #19564 from srowen/SPARK-21936.2.

(cherry picked from commit 8beeaed66bde0ace44495b38dc967816e16b3464)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/daa838b8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/daa838b8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/daa838b8

Branch: refs/heads/branch-2.2
Commit: daa838b8886496e64700b55d1301d348f1d5c9ae
Parents: bf8163f
Author: Sean Owen <sowen@cloudera.com>
Authored: Tue Oct 24 13:56:10 2017 +0100
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Tue Oct 24 13:56:31 2017 +0100

----------------------------------------------------------------------
 .../spark/sql/hive/HiveExternalCatalogVersionsSuite.scala      | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/daa838b8/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index 305f5b5..5f8c9d5 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -53,7 +53,9 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
   private def downloadSpark(version: String): Unit = {
     import scala.sys.process._
 
-    val url = s"https://d3kbcqa49mib13.cloudfront.net/spark-$version-bin-hadoop2.7.tgz"
+    val preferredMirror =
+      Seq("wget", "https://www.apache.org/dyn/closer.lua?preferred=true", "-q", "-O", "-").!!.trim
+    val url = s"$preferredMirror/spark/spark-$version/spark-$version-bin-hadoop2.7.tgz"
 
     Seq("wget", url, "-q", "-P", sparkTestingDir.getCanonicalPath).!
 
@@ -142,7 +144,7 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
 
 object PROCESS_TABLES extends QueryTest with SQLTestUtils {
   // Tests the latest version of every release line.
-  val testingVersions = Seq("2.0.2", "2.1.1", "2.2.0")
+  val testingVersions = Seq("2.0.2", "2.1.2", "2.2.0")
 
   protected var spark: SparkSession = _
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message