spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <ravishankar.n...@gmail.com>
Subject Re: Save hive table from spark in hive 2.1.0
Date Sun, 10 Dec 2017 15:21:26 GMT
Hi,

Good try. As you can see, when you run upgrade using schematool, there is a
duplicate column error. Can you please look the script generated and edit
to avoid duplicate column?

Not sure why the Hive guys made it complicated, I did face same issues like
you.

Can anyone else give a clean and better option?

Best, Ravion


On Dec 10, 2017 6:17 AM, "Alejandro Reina" <alex.konu@gmail.com> wrote:

> I have tried what you propose, added the property to hive-site.xml, and
> although with this option I can run hive, this does not solve my problem.
> I'm sorry if maybe you explain me wrongly.
>
> I need to save a dataframe transformed into spark in hive, with the version
> of scheme 2.1.1 of hive (last stable version of hive until 2 months ago),
> by
> default spark internally uses version 1.2.1 and this generates conflict
> with
> my other applications the ecosystem.
>
> This hiveTable created from spark, is not visible from any other
> application
> due to its version of the metadata, so I need to create from spark a hive
> table in its corresponding version.
>
> Hive schematool
> root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-
> log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL:        jdbc:mysql://localhost/
> metastore?useSSL=false
> Metastore Connection Driver :    com.mysql.jdbc.Driver
> Metastore connection User:       hive
> Hive distribution version:       2.1.0
> Metastore schema version:        1.2.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema
> version
> is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema
> version
> is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
>         at
> org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(
> HiveSchemaTool.java:202)
>         at org.apache.hive.beeline.HiveSchemaTool.showInfo(
> HiveSchemaTool.java:139)
>         at org.apache.hive.beeline.HiveSchemaTool.main(
> HiveSchemaTool.java:498)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> *** schemaTool failed ***
>
> If i try upgrade schema too have error.
> root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-
> log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL:        jdbc:mysql://localhost/
> metastore?useSSL=false
> Metastore Connection Driver :    com.mysql.jdbc.Driver
> Metastore connection User:       hive
> Starting upgrade metastore schema from version 1.2.0 to 2.1.0
> Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql
> Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060)
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED!
> Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> Use --verbose for detailed stacktrace.
> *** schemaTool failed ***
>
> thank you very much for your time dedicated to helping me
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message