Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 34391200D59 for ; Sun, 10 Dec 2017 16:21:35 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 2C58E160C09; Sun, 10 Dec 2017 15:21:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 23165160C00 for ; Sun, 10 Dec 2017 16:21:33 +0100 (CET) Received: (qmail 48111 invoked by uid 500); 10 Dec 2017 15:21:31 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 48101 invoked by uid 99); 10 Dec 2017 15:21:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Dec 2017 15:21:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5DD98180145 for ; Sun, 10 Dec 2017 15:21:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.192 X-Spam-Level: **** X-Spam-Status: No, score=4.192 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id bL7Di5xYsHpe for ; Sun, 10 Dec 2017 15:21:28 +0000 (UTC) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2026C5F2EE for ; Sun, 10 Dec 2017 15:21:28 +0000 (UTC) Received: by mail-lf0-f51.google.com with SMTP id 74so16534509lfs.0 for ; Sun, 10 Dec 2017 07:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Lt9TFkXeLST+kBz8OT61Rs26VIQG861I/jUH8CZJbQk=; b=fWKKgk0CNK76HZuuyCtRd9rWBMMPxI+83HWF9KfnYagX/OK5RKUudXGfrxeKTNE4EL tAp/cIBUHZ337sIqkMyjBjoJxtLnF6Q7kt4W2ECExxq5z7mWL7AIXwrd9XLX96pqz+H5 kMZdDbmLXNnQox9pKgFoyPuf6j2RUHGFlIL/S4sQBmMG28DYXoA/SPQOlVpdWWsyOY/7 SACOYfw+xD4or+j7hffBe5klCyhtiArvva4bylhENuZb6MhrwXc/0o2odw8AdJBp4kPP nr7OdrayHPhwHAixtn8es1OL3EA/1T23p+3ZpyXv+rcmy8UKlLsDbh8Prd3haIvDPiGV qYhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Lt9TFkXeLST+kBz8OT61Rs26VIQG861I/jUH8CZJbQk=; b=RHrbRIXcHUikYHuz4j+u0wG7lu5QGkrgVoah0qmZnu/KhD6uu6ihHwwTFT4E4GC1pt d5jTCbdLxYhcpOoED9kzB4vkvdPG8ASuEt8xNgFHLFgZFZm4hXI0vyFo7wRHUwP94q8g ywaS4VYFfwdIdrl+nnDtNsk5/wkHFyXY01Fh35mbWtSFWBfIPaisBfYco0LevntTn8v3 0X8cKKH4FIvZbXwCxl5kiEs3t0iTV8Xzdle/FSle63VB1YRzEUwYYCpfo5pDlHoCQ8V3 Dy54kCjH+WkHTkx2+b8XL+sq2SSDlwBtUfObKUX3MMestzb+IYSPseTJNqfUTHH/JEtV g5Rw== X-Gm-Message-State: AJaThX6u5j2z7hsYyM/ZIp2rE8Y6ZbHgWef5FEdQqt9pvu/H0MjAt8Q3 +9GoT8wdk/WUsDOiARzXcLVN9k9a1/soMV/Zs8Y= X-Google-Smtp-Source: AGs4zMaTpXnYNj8kCK+mJilHtcifCeguIKtwKZIcRtow/n/2yda+biJeFWFrDPM2gGzkfA73/EkC3KLSO4mxniMsOD4= X-Received: by 10.46.27.85 with SMTP id b82mr18723156ljb.163.1512919287419; Sun, 10 Dec 2017 07:21:27 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.59.15 with HTTP; Sun, 10 Dec 2017 07:21:26 -0800 (PST) Received: by 10.25.59.15 with HTTP; Sun, 10 Dec 2017 07:21:26 -0800 (PST) In-Reply-To: <1512904664425-0.post@n3.nabble.com> References: <1512860159617-0.post@n3.nabble.com> <1512904664425-0.post@n3.nabble.com> From: =?UTF-8?B?4pi8IFIgTmFpciAo4KSw4KS14KS/4KS24KSC4KSV4KSwIOCkqOCkvuCkr+CksCk=?= Date: Sun, 10 Dec 2017 10:21:26 -0500 Message-ID: Subject: Re: Save hive table from spark in hive 2.1.0 To: alex.konu@gmail.com, user@spark.apache.org Content-Type: multipart/alternative; boundary="94eb2c1a60fab355d5055ffdf851" archived-at: Sun, 10 Dec 2017 15:21:35 -0000 --94eb2c1a60fab355d5055ffdf851 Content-Type: text/plain; charset="UTF-8" Hi, Good try. As you can see, when you run upgrade using schematool, there is a duplicate column error. Can you please look the script generated and edit to avoid duplicate column? Not sure why the Hive guys made it complicated, I did face same issues like you. Can anyone else give a clean and better option? Best, Ravion On Dec 10, 2017 6:17 AM, "Alejandro Reina" wrote: > I have tried what you propose, added the property to hive-site.xml, and > although with this option I can run hive, this does not solve my problem. > I'm sorry if maybe you explain me wrongly. > > I need to save a dataframe transformed into spark in hive, with the version > of scheme 2.1.1 of hive (last stable version of hive until 2 months ago), > by > default spark internally uses version 1.2.1 and this generates conflict > with > my other applications the ecosystem. > > This hiveTable created from spark, is not visible from any other > application > due to its version of the metadata, so I need to create from spark a hive > table in its corresponding version. > > Hive schematool > root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j- > log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Metastore connection URL: jdbc:mysql://localhost/ > metastore?useSSL=false > Metastore Connection Driver : com.mysql.jdbc.Driver > Metastore connection User: hive > Hive distribution version: 2.1.0 > Metastore schema version: 1.2.0 > org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema > version > is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0 > org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema > version > is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0 > at > org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion( > HiveSchemaTool.java:202) > at org.apache.hive.beeline.HiveSchemaTool.showInfo( > HiveSchemaTool.java:139) > at org.apache.hive.beeline.HiveSchemaTool.main( > HiveSchemaTool.java:498) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > *** schemaTool failed *** > > If i try upgrade schema too have error. > root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j- > log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Metastore connection URL: jdbc:mysql://localhost/ > metastore?useSSL=false > Metastore Connection Driver : com.mysql.jdbc.Driver > Metastore connection User: hive > Starting upgrade metastore schema from version 1.2.0 to 2.1.0 > Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql > Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060) > org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! > Metastore state would be inconsistent !! > Underlying cause: java.io.IOException : Schema script failed, errorcode 2 > Use --verbose for detailed stacktrace. > *** schemaTool failed *** > > thank you very much for your time dedicated to helping me > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscribe@spark.apache.org > > --94eb2c1a60fab355d5055ffdf851 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Good try= . As you can see, when you run upgrade using schematool, there is a duplica= te column error. Can you please look the script generated and edit to avoid= duplicate column?

Not s= ure why the Hive guys made it complicated, I did face same issues like you.=

Can anyone else give a = clean and better option?

Best, Ravion


On Dec 10, 2017 6:17 AM, "Alejandr= o Reina" <alex.konu@gmail.co= m> wrote:
I h= ave tried what you propose, added the property to hive-site.xml, and
although with this option I can run hive, this does not solve my problem. I'm sorry if maybe you explain me wrongly.

I need to save a dataframe transformed into spark in hive, with the version=
of scheme 2.1.1 of hive (last stable version of hive until 2 months ago), b= y
default spark internally uses version 1.2.1 and this generates conflict wit= h
my other applications the ecosystem.

This hiveTable created from spark, is not visible from any other applicatio= n
due to its version of the metadata, so I need to create from spark a hive table in its corresponding version.

Hive schematool
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --ver= bose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf= 4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12= -1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#mul= tiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:=C2=A0 =C2=A0 =C2=A0 =C2=A0 jdbc:mysql://localhost= /metastore?useSSL=3Dfalse
Metastore Connection Driver :=C2=A0 =C2=A0 com.mysql.jdbc.Driver
Metastore connection User:=C2=A0 =C2=A0 =C2=A0 =C2=A0hive
Hive distribution version:=C2=A0 =C2=A0 =C2=A0 =C2=A02.1.0
Metastore schema version:=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema v= ersion
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema v= ersion
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at
org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(HiveSchemaTool.java:202)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hive.beeline.HiveSchemaTool.= showInfo(HiveSchemaTool.java:139)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hive.beeline.HiveSchemaTool.= main(HiveSchemaTool.java:498)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce= ssorImpl.java:62)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe= thodAccessorImpl.java:43)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.reflect.Method.invoke(Method.= java:498)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.util.RunJar.run(RunJa= r.java:221)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.util.RunJar.main(RunJ= ar.java:136)
*** schemaTool failed ***

If i try upgrade schema too have error.
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSch= ema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf= 4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12= -1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#mul= tiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:=C2=A0 =C2=A0 =C2=A0 =C2=A0 jdbc:mysql://localhost= /metastore?useSSL=3Dfalse
Metastore Connection Driver :=C2=A0 =C2=A0 com.mysql.jdbc.Driver
Metastore connection User:=C2=A0 =C2=A0 =C2=A0 =C2=A0hive
Starting upgrade metastore schema from version 1.2.0 to 2.1.0
Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql
Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=3D42S21,cod= e=3D1060)
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace.
*** schemaTool failed ***

thank you very much for your time dedicated to helping me




--
Sent from: http://apache-spark-user-list.10= 01560.n3.nabble.com/

-----------------------------------------------------------------= ----
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

--94eb2c1a60fab355d5055ffdf851--