Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0634011764 for ; Mon, 19 May 2014 23:53:41 +0000 (UTC) Received: (qmail 49563 invoked by uid 500); 19 May 2014 23:53:40 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 49497 invoked by uid 500); 19 May 2014 23:53:40 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 49483 invoked by uid 500); 19 May 2014 23:53:40 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 49467 invoked by uid 99); 19 May 2014 23:53:40 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 May 2014 23:53:40 +0000 Date: Mon, 19 May 2014 23:53:40 +0000 (UTC) From: "David Chen (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-7066) hive-exec jar is missing avro-mapred MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002601#comment-14002601 ] David Chen commented on HIVE-7066: ---------------------------------- [~xuefuz], were you able to reproduce the problem? Do you have any more feedback on this patch before it can be merged? > hive-exec jar is missing avro-mapred > ------------------------------------ > > Key: HIVE-7066 > URL: https://issues.apache.org/jira/browse/HIVE-7066 > Project: Hive > Issue Type: Bug > Reporter: David Chen > Assignee: David Chen > Attachments: HIVE-7066.1.patch > > > Running a simple query that reads an Avro table caused the following exception to be thrown on the cluster side: > {code} > java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat > Serialization trace: > outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) > aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) > at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365) > at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276) > at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254) > at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445) > at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438) > at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) > at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:191) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:394) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat > Serialization trace: > outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) > aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) > at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) > at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942) > at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850) > at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864) > at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334) > ... 13 more > Caused by: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat > at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45) > at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:56) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:476) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:148) > at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) > at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238) > at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226) > at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745) > at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) > ... 25 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:32) > ... 37 more > Caused by: java.lang.NoClassDefFoundError: org/apache/avro/io/DatumWriter > at java.lang.Class.getDeclaredFields0(Native Method) > at java.lang.Class.privateGetDeclaredFields(Class.java:2348) > at java.lang.Class.getDeclaredFields(Class.java:1779) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:150) > at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.(FieldSerializer.java:109) > ... 42 more > Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumWriter > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 47 more > {code} > I took a look at the hive-exec jar and found that the Avro core jar was not included, though avro-mapred is included. > I confirmed that Avro core was included in the Hive 0.12 hive-exec jar. Was there a reason why this was removed in trunk? It seems that this would break the AvroSerDe. -- This message was sent by Atlassian JIRA (v6.2#6252)