Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3229518BB5 for ; Sun, 10 Jan 2016 14:26:14 +0000 (UTC) Received: (qmail 10391 invoked by uid 500); 10 Jan 2016 14:26:10 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 10294 invoked by uid 500); 10 Jan 2016 14:26:10 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 10284 invoked by uid 99); 10 Jan 2016 14:26:10 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2016 14:26:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 04F25C2B03 for ; Sun, 10 Jan 2016 14:26:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.899 X-Spam-Level: ** X-Spam-Status: No, score=2.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id XjBNWFWKSEq6 for ; Sun, 10 Jan 2016 14:26:04 +0000 (UTC) Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 3A98A31AB5 for ; Sun, 10 Jan 2016 14:26:03 +0000 (UTC) Received: by mail-ig0-f172.google.com with SMTP id mw1so97980603igb.1 for ; Sun, 10 Jan 2016 06:26:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+i0/4Ocn90ApEfUx+czPlFFxZnsj19vO7k7nK5Y+3BM=; b=U7ogo89uay7GvCFAv2hfIpw6+tZa9Xn20B1ZY7XUm+lSmILphWw3RdfsZvK4dwT1mB eBasm5jXYCYeYvxxi0GO3bU6mtuWU+oYeQVDEqxZWtTSYiHFu7nLXkJRur1IdXnfIktU Z16TD1Weu79kitr76tm5pacHORGeMgJ59txe3K5P+oz31h9OKn2i3AuT+aUdO64SQIhf 1ILnZs8JHJJvlw6DPPkXrPqAhS4v8NaJ5tOKIO1iGy9MTOcOa/noEbNPvuxcmdHeTnty j8fhzepbDqjD6Ik77kc6uzaDXQzothJBB1o1R1/o7NbkW3ysL6lCpGm60fQPc9Ryk8xq m9+g== MIME-Version: 1.0 X-Received: by 10.50.136.226 with SMTP id qd2mr7935940igb.37.1452435962226; Sun, 10 Jan 2016 06:26:02 -0800 (PST) Received: by 10.107.144.198 with HTTP; Sun, 10 Jan 2016 06:26:02 -0800 (PST) In-Reply-To: References: Date: Sun, 10 Jan 2016 19:56:02 +0530 Message-ID: Subject: Re: adding jars - hive on spark cdh 5.4.3 From: sandeep vura To: Ophir Etzion Cc: user@hive.apache.org, Prem Sure , spark users Content-Type: multipart/alternative; boundary=089e013cba049671540528fb9997 --089e013cba049671540528fb9997 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Upgrade to CDH 5.5 for spark. It should work On Sat, Jan 9, 2016 at 12:17 AM, Ophir Etzion wrote: > It didn't work. assuming I did the right thing. > in the properties you could see > > {"key":"hive.aux.jars.path","value":"file:///data/loko/foursquare.web-hiv= erc/current/hadoop-hive-serde.jar,file:///data/loko/foursquare.web-hiverc/c= urrent/hadoop-hive-udf.jar","isFinal":false,"resource":"programatically"} > which includes the jar that has the class I need but I still get > > org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find c= lass: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat > > > > On Fri, Jan 8, 2016 at 12:24 PM, Edward Capriolo > wrote: > >> You can not 'add jar' input formats and serde's. They need to be part of >> your auxlib. >> >> On Fri, Jan 8, 2016 at 12:19 PM, Ophir Etzion >> wrote: >> >>> I tried now. still getting >>> >>> 16/01/08 16:37:34 ERROR exec.Utilities: Failed to load plan: hdfs://had= oop-alidoro-nn-vip/tmp/hive/hive/c2af9882-38a9-42b0-8d17-3f56708383e8/hive_= 2016-01-08_16-36-41_370_3307331506800215903-3/-mr-10004/3c90a796-47fc-4541-= bbec-b196c40aefab/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoEx= ception: Unable to find class: com.foursquare.hadoop.hive.io.HiveThriftSequ= enceFileInputFormat >>> Serialization trace: >>> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) >>> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) >>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find= class: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat >>> >>> >>> HiveThriftSequenceFileInputFormat is in one of the jars I'm trying to a= dd. >>> >>> >>> On Thu, Jan 7, 2016 at 9:58 PM, Prem Sure wrote= : >>> >>>> did you try -- jars property in spark submit? if your jar is of huge >>>> size, you can pre-load the jar on all executors in a common available >>>> directory to avoid network IO. >>>> >>>> On Thu, Jan 7, 2016 at 4:03 PM, Ophir Etzion >>>> wrote: >>>> >>>>> I' trying to add jars before running a query using hive on spark on >>>>> cdh 5.4.3. >>>>> I've tried applying the patch in >>>>> https://issues.apache.org/jira/browse/HIVE-12045 (manually as the >>>>> patch is done on a different hive version) but still hasn't succeeded= . >>>>> >>>>> did anyone manage to do ADD JAR successfully with CDH? >>>>> >>>>> Thanks, >>>>> Ophir >>>>> >>>> >>>> >>> >> > --089e013cba049671540528fb9997 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Upgrade to CDH 5.5 for spark. It should work

On Sat, Jan 9, 2016 at 12= :17 AM, Ophir Etzion <ophir@foursquare.com> wrote:
It didn't work. assuming I= did the right thing.
in the properties =C2=A0you could see
<= pre style=3D"margin-top:0px;margin-bottom:0px;border:0px">{"key":= "hive.aux.jars.path","value":"file:///data/loko/fo= ursquare.web-hiverc/current/hadoop-hive-serde.jar,file:///data/loko/foursqu= are.web-hiverc/current/hadoop-hive-udf.jar","isFinal":false,= "resource":"programatically"} which includes the jar that has the class I need but I still get 
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find cl=
ass: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat


On Fri, Jan 8, 2016 at 12:2= 4 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
=
You can not 'add jar= 9; input formats and serde's. They need to be part of your auxlib.

On Fri= , Jan 8, 2016 at 12:19 PM, Ophir Etzion <ophir@foursquare.com> wrote:
I tried now= . still getting

16/01/08 16:37:34 ERROR exec.Utilities: Failed to load plan: hdfs://=
hadoop-alidoro-nn-vip/tmp/hive/hive/c2af9882-38a9-42b0-8d17-3f56708383e8/hi=
ve_2016-01-08_16-36-41_370_3307331506800215903-3/-mr-10004/3c90a796-47fc-45=
41-bbec-b196c40aefab/map.xml: org.apache.hive.com.esotericsoftware.kryo.Kry=
oException: Unable to find class: com.foursquare.hadoop.hive.io.HiveThriftS=
equenceFileInputFormat
Serialization trace:
inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find cla=
ss: com.foursquare.hadoop.hive.io.HiveThriftSequenceFileInputFormat


HiveThriftSequen=
ceFileInputFormat is in one of the jars I'm trying to add.
<= /div>

On Thu, Jan = 7, 2016 at 9:58 PM, Prem Sure <premsure542@gmail.com> wr= ote:
did you try -- jars= property in spark submit? if your jar is of huge size, you can pre-load th= e jar on all executors in a common available directory to avoid network IO.=

On Thu, Jan 7, 2016 at 4:03 PM, Ophir Etzion <ophir@foursquare.com> wrote:
I&#= 39; trying to add jars before running a query using hive on spark on cdh 5.= 4.3.
I've tried applying the patch in=C2=A0https://issues.apach= e.org/jira/browse/HIVE-12045 (manually as the patch is done on a differ= ent hive version) but still hasn't succeeded.

did anyone manage = to do ADD JAR successfully with CDH?

Thanks,
Op= hir





--089e013cba049671540528fb9997--