Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0701410287 for ; Fri, 23 Jan 2015 06:43:38 +0000 (UTC) Received: (qmail 50656 invoked by uid 500); 23 Jan 2015 06:43:37 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 50577 invoked by uid 500); 23 Jan 2015 06:43:37 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 50467 invoked by uid 500); 23 Jan 2015 06:43:37 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 50449 invoked by uid 99); 23 Jan 2015 06:43:37 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jan 2015 06:43:37 +0000 Date: Fri, 23 Jan 2015 06:43:37 +0000 (UTC) From: "Chengxiang Li (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9410: -------------------------------- Attachment: HIVE-9410.4-spark.patch > ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch] > ---------------------------------------------------------------------------------------------- > > Key: HIVE-9410 > URL: https://issues.apache.org/jira/browse/HIVE-9410 > Project: Hive > Issue Type: Sub-task > Components: Spark > Environment: CentOS 6.5 > JDK1.7 > Reporter: Xin Hao > Assignee: Chengxiang Li > Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch, HIVE-9410.3-spark.patch, HIVE-9410.4-spark.patch > > > We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18 etc.). It will be passed for default Hive (on MR) mode, while failed for Hive On Spark mode (both Standalone and Yarn-Client). > Although we use 'add jar XXXX.jar;' to add the UDF jar explicitly, the issue still exists. > BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed. > Detail Error Message is as below (NOTE: de.bankmark.bigbench.queries.q10.SentimentUDF is the UDF which contained in jar bigbenchqueriesmr.jar, and we have add command like 'add jar /location/to/bigbenchqueriesmr.jar;' into .sql explicitly) > INFO [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) - Failed to run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d > org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: de.bankmark.bigbench.queries.q10.SentimentUDF > Serialization trace: > genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc) > conf (org.apache.hadoop.hive.ql.exec.UDTFOperator) > childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator) > childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator) > childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) > aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) > right (org.apache.commons.lang3.tuple.ImmutablePair) > edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > ... > Caused by: java.lang.ClassNotFoundException: de.bankmark.bigbench.queries.q10.SentimentUDF > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136) > ... 55 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)