spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-14204][SQL] register driverClass rather than user-specified class
Date Wed, 03 Aug 2016 20:15:16 GMT
Repository: spark
Updated Branches:
  refs/heads/master e6f226c56 -> 685b08e26


[SPARK-14204][SQL] register driverClass rather than user-specified class

This is a pull request that was originally merged against branch-1.6 as #12000, now being
merged into master as well.  srowen zzcclp JoshRosen

This pull request fixes an issue in which cluster-mode executors fail to properly register
a JDBC driver when the driver is provided in a jar by the user, but the driver class name
is derived from a JDBC URL (rather than specified by the user). The consequence of this is
that all JDBC accesses under the described circumstances fail with an IllegalStateException.
I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204

My proposed solution is to have the executors register the JDBC driver class under all circumstances,
not only when the driver is specified by the user.

This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed
that the problem was fixed.

Author: Kevin McHale <kevin@premise.com>

Closes #14420 from mchalek/mchalek-jdbc_driver_registration.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/685b08e2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/685b08e2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/685b08e2

Branch: refs/heads/master
Commit: 685b08e2611b69f8db60a00c0c94aecd315e2a3e
Parents: e6f226c
Author: Kevin McHale <kevin@premise.com>
Authored: Wed Aug 3 13:15:13 2016 -0700
Committer: Sean Owen <sowen@cloudera.com>
Committed: Wed Aug 3 13:15:13 2016 -0700

----------------------------------------------------------------------
 .../apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/685b08e2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
index 81d38e3..a33c26d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
@@ -55,7 +55,7 @@ object JdbcUtils extends Logging {
       DriverManager.getDriver(url).getClass.getCanonicalName
     }
     () => {
-      userSpecifiedDriverClass.foreach(DriverRegistry.register)
+      DriverRegistry.register(driverClass)
       val driver: Driver = DriverManager.getDrivers.asScala.collectFirst {
         case d: DriverWrapper if d.wrapped.getClass.getCanonicalName == driverClass =>
d
         case d if d.getClass.getCanonicalName == driverClass => d


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message