spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-17355] Workaround for HIVE-14684 / HiveResultSetMetaData.isSigned exception
Date Thu, 01 Sep 2016 23:45:38 GMT
Repository: spark
Updated Branches:
  refs/heads/master d314677cf -> 15539e54c


[SPARK-17355] Workaround for HIVE-14684 / HiveResultSetMetaData.isSigned exception

## What changes were proposed in this pull request?

Attempting to use Spark SQL's JDBC data source against the Hive ThriftServer results in a
`java.sql.SQLException: Method` not supported exception from `org.apache.hive.jdbc.HiveResultSetMetaData.isSigned`.
Here are two user reports of this issue:

- https://stackoverflow.com/questions/34067686/spark-1-5-1-not-working-with-hive-jdbc-1-2-0
- https://stackoverflow.com/questions/32195946/method-not-supported-in-spark

I have filed [HIVE-14684](https://issues.apache.org/jira/browse/HIVE-14684) to attempt to
fix this in Hive by implementing the isSigned method, but in the meantime / for compatibility
with older JDBC drivers I think we should add special-case error handling to work around this
bug.

This patch updates `JDBCRDD`'s `ResultSetMetadata` to schema conversion to catch the "Method
not supported" exception from Hive and return `isSigned = true`. I believe that this is safe
because, as far as I know, Hive does not support unsigned numeric types.

## How was this patch tested?

Tested manually against a Spark Thrift Server.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #14911 from JoshRosen/hive-jdbc-workaround.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/15539e54
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/15539e54
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/15539e54

Branch: refs/heads/master
Commit: 15539e54c2650a164f09c072f8fae934bb0468c9
Parents: d314677
Author: Josh Rosen <joshrosen@databricks.com>
Authored: Thu Sep 1 16:45:26 2016 -0700
Committer: Josh Rosen <joshrosen@databricks.com>
Committed: Thu Sep 1 16:45:26 2016 -0700

----------------------------------------------------------------------
 .../spark/sql/execution/datasources/jdbc/JDBCRDD.scala   | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/15539e54/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
index 8d9048a..9b5088f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
@@ -136,7 +136,16 @@ object JDBCRDD extends Logging {
             val typeName = rsmd.getColumnTypeName(i + 1)
             val fieldSize = rsmd.getPrecision(i + 1)
             val fieldScale = rsmd.getScale(i + 1)
-            val isSigned = rsmd.isSigned(i + 1)
+            val isSigned = {
+              try {
+                rsmd.isSigned(i + 1)
+              } catch {
+                // Workaround for HIVE-14684:
+                case e: SQLException if
+                  e.getMessage == "Method not supported" &&
+                  rsmd.getClass.getName == "org.apache.hive.jdbc.HiveResultSetMetaData" =>
true
+              }
+            }
             val nullable = rsmd.isNullable(i + 1) != ResultSetMetaData.columnNoNulls
             val metadata = new MetadataBuilder()
               .putString("name", columnName)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message