Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7E9CA200827 for ; Sun, 15 May 2016 08:32:04 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 712FC1609AD; Sun, 15 May 2016 06:32:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 681AF16098E for ; Sun, 15 May 2016 08:32:03 +0200 (CEST) Received: (qmail 38119 invoked by uid 500); 15 May 2016 06:32:01 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 38109 invoked by uid 99); 15 May 2016 06:32:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2016 06:32:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8918FC0227 for ; Sun, 15 May 2016 06:32:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id CUS95BLH7i_F for ; Sun, 15 May 2016 06:31:57 +0000 (UTC) Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 24DA95FAFA for ; Sun, 15 May 2016 06:31:57 +0000 (UTC) Received: by mail-qk0-f178.google.com with SMTP id x7so80919495qkd.3 for ; Sat, 14 May 2016 23:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=KriPKQE+zNgbWkLf34vkOhEfGZJRU8BsPaI2fXzWXWQ=; b=05Yw9p7b7iwT3GsLOtZmMTp30H/h8iUFka2dkJ8drs/DxlRa2H8qj7NHnLcGNYmNHV Nj9RXK4B/+MbHotHKotsG+aK5msrXBoucWOF/Dp5dglqf/TUuGsFmsmuLPey6A0ylwUh 9qdJ8s0W3rw+xpp/HU00f17sXZ/5duS7OnO2AeDsujx2kgcJbQpIYfFzEj7kTjAVpQKw 7qU3wTAt8MMLlvuRhexnYtpjJPWrGeBbqm3RG1v5601nj4dXufHG9CsYGO+R5jIAmtjc nWdY4BiTmtPzPq/jVKtxLKRNyJpuiOjPhWbGTPN7z15rjtd0ym7BN1cXavvdpBJ1KDiB hGng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=KriPKQE+zNgbWkLf34vkOhEfGZJRU8BsPaI2fXzWXWQ=; b=MURVnppflLcKHRZYFnoUzd6auzbkrxSvR8r/lSpXRI1I4Z4/9usppIMHTWVNizciZC T87QJ2og10JrZtqwH1aLRzNMkvC9ZlQXVaMnTYqAE9pinIfsIB8lJqKshXkOu0ALdGYa s/SyQc0Fa7mNg9elDCI9O+VPEfqni+JS4yw3OVdPc4ohzR5fQBxP9NcszFaxqkRFqM0Y kJVRTAlABzAJvEZrJztjb1GOUnebnzTmGbZEPbjJNID2Zz9ILF/IV+sPuuG/u13879sS lF7v76rMH4kjryI3aEM1CVVKkDNfWE0YZgkU6ooqX6uoagHm40LANj5ik34cLKHsThJC BBwg== X-Gm-Message-State: AOPr4FXJ9vAv8pDxEllfring++1q/2Y4MsdOwxYDl1IvS50V0ma/JOQ1Rt9RAeomEDPbHbKwVSSWaA8hISVOdA== MIME-Version: 1.0 X-Received: by 10.55.20.167 with SMTP id 39mr17946398qku.197.1463293916166; Sat, 14 May 2016 23:31:56 -0700 (PDT) Received: by 10.55.70.69 with HTTP; Sat, 14 May 2016 23:31:56 -0700 (PDT) In-Reply-To: References: Date: Sun, 15 May 2016 07:31:56 +0100 Message-ID: Subject: Re: spark sql write orc table on viewFS throws exception From: Mich Talebzadeh To: linxi zeng Cc: "user @spark" Content-Type: multipart/alternative; boundary=001a113ad6b41342080532dbaaaf archived-at: Sun, 15 May 2016 06:32:04 -0000 --001a113ad6b41342080532dbaaaf Content-Type: text/plain; charset=UTF-8 I am not sure this is going to resolve INSERT OVEERWRITE into ORC table issue. Can you go to hive and do show create table custom.rank_less_orc_none and send the output. Is that table defined as transactional? Other alternative is to use Spark to insert into a normal text table and do insert from the text table into ORC using HiveContext or doing it purely in Hive Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 15 May 2016 at 04:01, linxi zeng wrote: > hi, all: > Recently, we have encountered a problem while using spark sql to write orc > table, which is related to > https://issues.apache.org/jira/browse/HIVE-10790. > In order to fix this problem we decided to patched the PR to the hive > branch which spark1.5 rely on. > We pull the hive branch( > https://github.com/pwendell/hive/tree/release-1.2.1-spark) and compile it > with cmd: mvn clean package -Phadoop-2,dist -DskipTests, and then upload to > the nexus without any problem. > > But when we compile spark with hive (group: org.spark-project.hive, > version: 1.2.1.spark) using cmd: ./make-distribution.sh --tgz -Phive > -Phive-thriftserver -Psparkr -Pyarn -Dhadoop.version=2.4.1 > -Dprotobuf.version=2.5.0 -DskipTests > we get this error msg: > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > spark-hive_2.10 --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 1 resource > [INFO] Copying 3 resources > [INFO][INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ > spark-hive_2.10 --- > [INFO] Using zinc server for incremental compilation > [info] Compiling 27 Scala sources and 1 Java source to > /home/sankuai/zenglinxi/spark/sql/hive/target/scala-2.10/classes... > [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found - > continuing with a stub. > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:35: > object shims is not a member of package org.apache.hadoop.hive > [error] import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader} > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:114: > not found: value ShimLoader > [error] val loadedShimsClassName = > ShimLoader.getHadoopShims.getClass.getCanonicalName > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:123: > not found: type ShimLoader > [error] val shimsField = > classOf[ShimLoader].getDeclaredField("hadoopShims") > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:127: > not found: type HadoopShims > [error] val shims = > classOf[HadoopShims].cast(shimsClass.newInstance()) > [error] ^ > [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not > found - continuing with a stub. > [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not > found - continuing with a stub. > [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found - > continuing with a stub. > [warn] four warnings found > [error] four errors found > [error] Compile failed at 2016-5-13 16:34:44 [4.348s] > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Spark Project Parent POM ........................... SUCCESS [ > 3.105 s] > [INFO] Spark Project Launcher ............................. SUCCESS [ > 8.360 s] > [INFO] Spark Project Networking ........................... SUCCESS [ > 8.491 s] > [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ > 5.110 s] > [INFO] Spark Project Unsafe ............................... SUCCESS [ > 6.854 s] > [INFO] Spark Project Core ................................. SUCCESS [02:33 > min] > [INFO] Spark Project Bagel ................................ SUCCESS [ > 5.183 s] > [INFO] Spark Project GraphX ............................... SUCCESS [ > 15.744 s] > [INFO] Spark Project Streaming ............................ SUCCESS [ > 39.070 s] > [INFO] Spark Project Catalyst ............................. SUCCESS [ > 57.416 s] > [INFO] Spark Project SQL .................................. SUCCESS [01:11 > min] > [INFO] Spark Project ML Library ........................... SUCCESS [01:28 > min] > [INFO] Spark Project Tools ................................ SUCCESS [ > 2.539 s] > [INFO] Spark Project Hive ................................. FAILURE [ > 13.273 s] > [INFO] Spark Project REPL ................................. SKIPPED > [INFO] Spark Project YARN ................................. SKIPPED > > This error information does throw doubt on > https://issues.apache.org/jira/browse/HIVE-10790. > Any one meet the same question with me? > --001a113ad6b41342080532dbaaaf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I am not sure this is going to resolve INSERT OVEERWR= ITE into ORC table issue. Can you go to hive and do

show create table custom.rank_less_orc_none

and= send the output.

Is that table defined as transac= tional?

Other alternative is to use Spark to inser= t into a normal text table and do insert from the text table into ORC using= HiveContext or doing it purely in Hive


On 15 May 2016 at 04:01, linxi zeng <= linxizeng0615@gmail.com> wrote:
hi, all:
Recently, we have encountered a problem = while=C2=A0using spark sql to write orc table, which is related to htt= ps://issues.apache.org/jira/browse/HIVE-10790.
In order to fi= x this problem we=C2=A0decided to=C2=A0patched the PR to the hive branch whi= ch spark1.5 rely on.
We pull the hive branch(https:/= /github.com/pwendell/hive/tree/release-1.2.1-spark) and compile it with= cmd:=C2=A0mvn clean package -Phadoop-2,dist -DskipTests,=C2=A0and then upl= oad to the nexus without any problem.
But when we compile spark with hive (group:=C2=A0org.spark= -project.hive, version:=C2=A01.2.1.spark) using cmd:=C2= =A0./make-distribution.sh --tgz -Phive -Phive-thriftserver -P= sparkr -Pyarn -Dhadoop.version=3D2.4.1 -Dprotobuf.version=3D2.5.0 -DskipTes= ts
we get this error msg:

[INFO] --- maven-= resources-plugin:2.6:resources (default-resources) @ spark-hive_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO][INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-fir= st) @ spark-hive_2.10 ---
[INFO] Using zinc server for incrementa= l compilation
[info] Compiling 27 Scala sources and 1 Java source= to /home/sankuai/zenglinxi/spark/sql/hive/target/scala-2.10/classes...
[warn] Class org.apache.hadoop.hive.shims.HadoopShims not found - co= ntinuing with a stub.
[error] /home/sankuai/zenglinxi/spark/sql/h= ive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:35:= object shims is not a member of package org.apache.hadoop.hive
[= error] import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader}
<= div>[error] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^
[error] /home/sankuai= /zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/C= lientWrapper.scala:114: not found: value ShimLoader
[error] =C2= =A0 =C2=A0 val loadedShimsClassName =3D ShimLoader.getHadoopShims.getClass.= getCanonicalName
[error] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^<= /div>
[error] /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org= /apache/spark/sql/hive/client/ClientWrapper.scala:123: not found: type Shim= Loader
[error] =C2=A0 =C2=A0 =C2=A0 val shimsField =3D classOf[Sh= imLoader].getDeclaredField("hadoopShims")
[error] =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^
[error] /home/sankuai/zenglin= xi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWra= pper.scala:127: not found: type HadoopShims
[error] =C2=A0 =C2=A0= =C2=A0 val shims =3D classOf[HadoopShims].cast(shimsClass.newInstance())
[error] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^
[warn] Class org.apache.hado= op.hive.thrift.HadoopThriftAuthBridge not found - continuing with a stub.
[warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge = not found - continuing with a stub.
[warn] Class org.apache.hadoo= p.hive.shims.HadoopShims not found - continuing with a stub.
[war= n] four warnings found
[error] four errors found
[error= ] Compile failed at 2016-5-13 16:34:44 [4.348s]
[INFO] ----------= --------------------------------------------------------------
[I= NFO] Reactor Summary:
[INFO]
[INFO] Spark Project Paren= t POM ........................... SUCCESS [ =C2=A03.105 s]
[INFO]= Spark Project Launcher ............................. SUCCESS [ =C2=A08.360= s]
[INFO] Spark Project Networking ........................... S= UCCESS [ =C2=A08.491 s]
[INFO] Spark Project Shuffle Streaming Se= rvice ............ SUCCESS [ =C2=A05.110 s]
[INFO] Spark Project = Unsafe ............................... SUCCESS [ =C2=A06.854 s]
[= INFO] Spark Project Core ................................. SUCCESS [02:33 m= in]
[INFO] Spark Project Bagel ................................ S= UCCESS [ =C2=A05.183 s]
[INFO] Spark Project GraphX .............= .................. SUCCESS [ 15.744 s]
[INFO] Spark Project Strea= ming ............................ SUCCESS [ 39.070 s]
[INFO] Spar= k Project Catalyst ............................. SUCCESS [ 57.416 s]
<= div>[INFO] Spark Project SQL .................................. SUCCESS [01= :11 min]
[INFO] Spark Project ML Library ........................= ... SUCCESS [01:28 min]
[INFO] Spark Project Tools ..............= .................. SUCCESS [ =C2=A02.539 s]
[INFO] Spark Project = Hive ................................. FAILURE [ 13.273 s]
[INFO]= Spark Project REPL ................................. SKIPPED
[IN= FO] Spark Project YARN ................................. SKIPPED

https://issu= es.apache.org/jira/browse/HIVE-10790.

Any one meet the same q= uestion with me?

--001a113ad6b41342080532dbaaaf--