spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tavis Barr (JIRA)" <>
Subject [jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore
Date Thu, 27 Sep 2018 14:02:00 GMT


Tavis Barr commented on SPARK-18112:

Why are you even asking these questions?  I have already pointed to the offending lines of
code in /src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala that are causing this error
and explained why the error is happening (see my comments from April 23rd).  All you have
to do is remove those two parameters and push.

> Spark2.x does not support read data from Hive 2.x metastore
> -----------------------------------------------------------
>                 Key: SPARK-18112
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: KaiXu
>            Assignee: Xiao Li
>            Priority: Critical
>             Fix For: 2.2.0
> Hive2.0 has been released in February 2016, after that Hive2.0.1 and Hive2.1.0 have also
been released for a long time, but till now spark only support to read hive metastore data
from Hive1.2.1 and older version, since Hive2.x has many bugs fixed and performance improvement
it's better and urgent to upgrade to support Hive2.x
> failed to load data from hive2.x metastore:
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
>         at org.apache.spark.sql.hive.HiveUtils$.hiveClientConfigurations(HiveUtils.scala:197)
>         at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
>         at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)
>         at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)
>         at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:4
>         at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)
>         at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50)
>         at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
>         at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:31)
>         at org.apache.spark.sql.SparkSession.table(SparkSession.scala:568)
>         at org.apache.spark.sql.SparkSession.table(SparkSession.scala:564)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message