spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Campagnola, Francesco" <Francesco.Campagn...@anritsu.com>
Subject RE: Spark 2.0.0 Thrift Server problem with Hive metastore
Date Tue, 06 Sep 2016 07:46:00 GMT
I mean I have installed Spark 2.0 in the same environment where Spark 1.6 thrift server was
running,
then stopped the Spark 1.6 thrift server and started the Spark 2.0 one.

If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and no upgrade
procedures are required.
The spark-defaults.conf file has not been changed.

The following commands issued to the Spark 2.0 thrift server work:
create database test;
use test;
create table tb_1 (id int);
insert into table tb_1 select t.id from (select 1 as id) t;

While all of these commands return the same error:
show databases;
show tables;
show partitions tb_1;
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage
62.0 failed 10 times, most recent failure: Lost task 0.9 in stage 62.0 (TID 540, vertica204):
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow
cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow




From: Jeff Zhang [mailto:zjffdu@gmail.com]
Sent: martedì 6 settembre 2016 02:50
To: Campagnola, Francesco <Francesco.Campagnola@anritsu.com>
Cc: user@spark.apache.org
Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore

How do you upgrade to spark 2.0 ?

On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco <Francesco.Campagnola@anritsu.com<mailto:Francesco.Campagnola@anritsu.com>>
wrote:
Hi,

in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, with Hive metastore
configured on Postgres DB, I have upgraded Spark to the 2.0.0.

I have started the thrift server on YARN, then tried to execute from the beeline cli or a
jdbc client the following command:
SHOW DATABASES;
It always gives this error on Spark server side:

spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):10000 -n spark

Connecting to jdbc:hive2://spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000
16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri:
jdbc:hive2:// spark-test:10000
Connected to: Spark SQL (version 2.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive

0: jdbc:hive2:// spark-test:10000> show databases;
java.lang.IllegalStateException: Can't overwrite cause with java.lang.ClassCastException:
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
        at java.lang.Throwable.initCause(Throwable.java:457)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
        at org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
        at org.apache.hive.service.cli.HiveSQLException.<init>(HiveSQLException.java:108)
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
        at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
        at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42)
        at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
        at org.apache.hive.beeline.Commands.execute(Commands.java:860)
        at org.apache.hive.beeline.Commands.sql(Commands.java:713)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage
3.0 failed 10 times, most recent failure: Lost task 0.9 in stage 3.0 (TID 12, vertica204):
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow
cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
        at org.apache.spark.scheduler.Task.run(Task.scala:85)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:244)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:210)
        ... 15 more
Error: Error retrieving next row (state=,code=0)

The same command works when using Spark 1.6, is it a possible issue?

Thanks!



--
Best Regards

Jeff Zhang
Mime
View raw message