spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: spark sql write orc table on viewFS throws exception
Date Sun, 15 May 2016 06:31:56 GMT
I am not sure this is going to resolve INSERT OVEERWRITE into ORC table
issue. Can you go to hive and do

show create table custom.rank_less_orc_none

and send the output.

Is that table defined as transactional?

Other alternative is to use Spark to insert into a normal text table and do
insert from the text table into ORC using HiveContext or doing it purely in
Hive


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 15 May 2016 at 04:01, linxi zeng <linxizeng0615@gmail.com> wrote:

> hi, all:
> Recently, we have encountered a problem while using spark sql to write orc
> table, which is related to
> https://issues.apache.org/jira/browse/HIVE-10790.
> In order to fix this problem we decided to patched the PR to the hive
> branch which spark1.5 rely on.
> We pull the hive branch(
> https://github.com/pwendell/hive/tree/release-1.2.1-spark) and compile it
> with cmd: mvn clean package -Phadoop-2,dist -DskipTests, and then upload to
> the nexus without any problem.
> <https://github.com/pwendell/hive/tree/release-1.2.1-spark>
> But when we compile spark with hive (group: org.spark-project.hive,
> version: 1.2.1.spark) using cmd: ./make-distribution.sh --tgz -Phive
> -Phive-thriftserver -Psparkr -Pyarn -Dhadoop.version=2.4.1
> -Dprotobuf.version=2.5.0 -DskipTests
> we get this error msg:
>
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> spark-hive_2.10 ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 1 resource
> [INFO] Copying 3 resources
> [INFO][INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-hive_2.10 ---
> [INFO] Using zinc server for incremental compilation
> [info] Compiling 27 Scala sources and 1 Java source to
> /home/sankuai/zenglinxi/spark/sql/hive/target/scala-2.10/classes...
> [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found -
> continuing with a stub.
> [error]
> /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:35:
> object shims is not a member of package org.apache.hadoop.hive
> [error] import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader}
> [error]                               ^
> [error]
> /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:114:
> not found: value ShimLoader
> [error]     val loadedShimsClassName =
> ShimLoader.getHadoopShims.getClass.getCanonicalName
> [error]                                ^
> [error]
> /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:123:
> not found: type ShimLoader
> [error]       val shimsField =
> classOf[ShimLoader].getDeclaredField("hadoopShims")
> [error]                                ^
> [error]
> /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:127:
> not found: type HadoopShims
> [error]       val shims =
> classOf[HadoopShims].cast(shimsClass.newInstance())
> [error]                           ^
> [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not
> found - continuing with a stub.
> [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not
> found - continuing with a stub.
> [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found -
> continuing with a stub.
> [warn] four warnings found
> [error] four errors found
> [error] Compile failed at 2016-5-13 16:34:44 [4.348s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [
>  3.105 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
>  8.360 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
>  8.491 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>  5.110 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
>  6.854 s]
> [INFO] Spark Project Core ................................. SUCCESS [02:33
> min]
> [INFO] Spark Project Bagel ................................ SUCCESS [
>  5.183 s]
> [INFO] Spark Project GraphX ............................... SUCCESS [
> 15.744 s]
> [INFO] Spark Project Streaming ............................ SUCCESS [
> 39.070 s]
> [INFO] Spark Project Catalyst ............................. SUCCESS [
> 57.416 s]
> [INFO] Spark Project SQL .................................. SUCCESS [01:11
> min]
> [INFO] Spark Project ML Library ........................... SUCCESS [01:28
> min]
> [INFO] Spark Project Tools ................................ SUCCESS [
>  2.539 s]
> [INFO] Spark Project Hive ................................. FAILURE [
> 13.273 s]
> [INFO] Spark Project REPL ................................. SKIPPED
> [INFO] Spark Project YARN ................................. SKIPPED
>
> This error information does throw doubt on
> https://issues.apache.org/jira/browse/HIVE-10790.
> Any one meet the same question with me?
>

Mime
View raw message