spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yh...@apache.org
Subject spark git commit: [SPARK-11614][SQL] serde parameters should be set only when all params are ready
Date Thu, 19 Nov 2015 03:37:18 GMT
Repository: spark
Updated Branches:
  refs/heads/master 67c75828f -> fc3f77b42


[SPARK-11614][SQL] serde parameters should be set only when all params are ready

see HIVE-7975 and HIVE-12373

With changed semantic of setters in thrift objects in hive, setter should be called only after
all parameters are set. It's not problem of current state but will be a problem in some day.

Author: navis.ryu <navis@apache.org>

Closes #9580 from navis/SPARK-11614.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fc3f77b4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fc3f77b4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fc3f77b4

Branch: refs/heads/master
Commit: fc3f77b42d62ca789d0ee07403795978961991c7
Parents: 67c7582
Author: navis.ryu <navis@apache.org>
Authored: Wed Nov 18 19:37:14 2015 -0800
Committer: Yin Huai <yhuai@databricks.com>
Committed: Wed Nov 18 19:37:14 2015 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala    | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/fc3f77b4/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
index f4d4571..9a981d0 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
@@ -804,12 +804,13 @@ private[hive] case class MetastoreRelation
 
       val serdeInfo = new org.apache.hadoop.hive.metastore.api.SerDeInfo
       sd.setSerdeInfo(serdeInfo)
+      // maps and lists should be set only after all elements are ready (see HIVE-7975)
       serdeInfo.setSerializationLib(p.storage.serde)
 
       val serdeParameters = new java.util.HashMap[String, String]()
-      serdeInfo.setParameters(serdeParameters)
       table.serdeProperties.foreach { case (k, v) => serdeParameters.put(k, v) }
       p.storage.serdeProperties.foreach { case (k, v) => serdeParameters.put(k, v) }
+      serdeInfo.setParameters(serdeParameters)
 
       new Partition(hiveQlTable, tPartition)
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message