spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-26421) Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use,just in idea local mode
Date Sun, 23 Dec 2018 00:22:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun resolved SPARK-26421.
-----------------------------------
          Resolution: Duplicate
    Target Version/s:   (was: 2.4.0)

> Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use,just in idea local mode
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-26421
>                 URL: https://issues.apache.org/jira/browse/SPARK-26421
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 2.4.0
>         Environment: ).idea maven project
> ).jdk 1.8.0_191
> ).hadoop 3.1.1
> ).spark 2.4.0
>            Reporter: thinktothings
>            Priority: Major
>
> ).Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use,just in idea local
mode
> ).idea maven project
> ).spark.sql connect hive 
> val spark = SparkSession
>  .builder()
>  .master("local")
>  .appName("Spark Hive Example")
>  .config("spark.sql.warehouse.dir", warehouseLocation)
>  .enableHiveSupport()
>  .getOrCreate()
> spark.sql("show databases").show()
>  
> ).do this   error ,local environment not cluser
> ----------------------
> Exception in thread "main" java.lang.ExceptionInInitializerError
>  at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
>  at java.lang.Class.forName0(Native Method)
>  at java.lang.Class.forName(Class.java:348)
>  at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
>  at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
>  at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
>  at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run$.main(Run.scala:19)
>  at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run.main(Run.scala)
> Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number:
3.1.1
>  at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
>  at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
>  at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
>  at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
>  ... 8 more
> Process finished with exit code 1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message