spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hvanhov...@apache.org
Subject spark git commit: [SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int
Date Mon, 12 Dec 2016 22:39:19 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.1 35011608f -> 523071f3f


[SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int

## What changes were proposed in this pull request?

Cloudera put `/var/run/cloudera-scm-agent/process/15000-hive-HIVEMETASTORE/hive-site.xml`
as the configuration file for the Hive Metastore Server, where `hive.metastore.try.direct.sql=false`.
But Spark isn't reading this configuration file and get default value `hive.metastore.try.direct.sql=true`.
As mallman said, we should use `getMetaConf` method to obtain the original configuration from
Hive Metastore Server. I have tested this method few times and the return value is always
consistent with Hive Metastore Server.

## How was this patch tested?

The existing tests.

Author: Yuming Wang <wgyumg@gmail.com>

Closes #16122 from wangyum/SPARK-18681.

(cherry picked from commit 90abfd15f4b3f612a7b0ff65f03bf319c78a0243)
Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/523071f3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/523071f3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/523071f3

Branch: refs/heads/branch-2.1
Commit: 523071f3fae72909b64c7f405868bbc85f5c3cde
Parents: 3501160
Author: Yuming Wang <wgyumg@gmail.com>
Authored: Mon Dec 12 23:38:36 2016 +0100
Committer: Herman van Hovell <hvanhovell@databricks.com>
Committed: Mon Dec 12 23:39:13 2016 +0100

----------------------------------------------------------------------
 .../scala/org/apache/spark/sql/hive/client/HiveShim.scala     | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/523071f3/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
index e561706..87f58e5 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
@@ -590,8 +590,11 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
       } else {
         logDebug(s"Hive metastore filter is '$filter'.")
         val tryDirectSqlConfVar = HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL
-        val tryDirectSql =
-          hive.getConf.getBoolean(tryDirectSqlConfVar.varname, tryDirectSqlConfVar.defaultBoolVal)
+        // We should get this config value from the metaStore. otherwise hit SPARK-18681.
+        // To be compatible with hive-0.12 and hive-0.13, In the future we can achieve this
by:
+        // val tryDirectSql = hive.getMetaConf(tryDirectSqlConfVar.varname).toBoolean
+        val tryDirectSql = hive.getMSC.getConfigValue(tryDirectSqlConfVar.varname,
+          tryDirectSqlConfVar.defaultBoolVal.toString).toBoolean
         try {
           // Hive may throw an exception when calling this method in some circumstances,
such as
           // when filtering on a non-string partition column when the hive config key


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message