hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: difference between add jar in hive session and hive --auxpath
Date Sat, 09 Mar 2013 17:58:08 GMT
BTW if your looking for protobuf + hive you should look at...


https://github.com/edwardcapriolo/hive-protobuf/

On Sat, Mar 9, 2013 at 12:32 PM, Ramki Palle <ramki.palle@gmail.com> wrote:

> I am not sure whether it works but try this:
>
> Put your ADD JAR commands into a file and invoke hive with -i file option.
>
> or
>
> insert your ADD JAR commands in your $HOME/.hiverc file and start hive.
>
>
> -Ramki.
>
>
> On Fri, Mar 8, 2013 at 11:55 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>
>> Essentially anything that is part of the InputFormat needs to be in
>> auxlib/auxpath. Anything part of a UDF can be added with 'add jar'.
>>
>>
>> On Fri, Mar 8, 2013 at 1:01 PM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:
>>
>>> --auxpath adds more jars to Hive's classpath before invoking Hive. ADD
>>> JARS copies jars around the cluster and adds them to the task classpath, so
>>> the jars you add aren't visible to hive itself. Annoying, but...
>>>
>>> On Fri, Mar 8, 2013 at 11:53 AM, java8964 java8964 <java8964@hotmail.com
>>> > wrote:
>>>
>>>>  This is in HIVE-0.9.0
>>>>
>>>> hive> list jars;
>>>> /nfs_home/common/userlibs/google-collections-1.0.jar
>>>> /nfs_home/common/userlibs/elephant-bird-hive-3.0.7.jar
>>>> /nfs_home/common/userlibs/protobuf-java-2.3.0.jar
>>>> /nfs_home/common/userlibs/elephant-bird-core-3.0.7.jar
>>>> file:/usr/lib/hive/lib/hive-builtins-0.9.0-cdh4.1.2.jar
>>>> hive> desc table;
>>>> java.lang.NoClassDefFoundError:
>>>> com/twitter/elephantbird/mapreduce/io/ProtobufConverter
>>>>         at
>>>> com.twitter.elephantbird.hive.serde.ProtobufDeserializer.initialize(ProtobufDeserializer.java:45)
>>>>         at
>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:203)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:930)
>>>>         at
>>>> org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:844)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2545)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:309)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>>>>         at
>>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>>>>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
>>>>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
>>>>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
>>>>         at
>>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
>>>>         at
>>>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
>>>>         at
>>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
>>>>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
>>>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>         at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> com.twitter.elephantbird.mapreduce.io.ProtobufConverter
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>         ... 25 more
>>>> FAILED: Execution Error, return code -101 from
>>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>>> hive> exit;
>>>> [y130zhan@daca2 userlibs]$ jar tvf
>>>> /nfs_home/common/userlibs/elephant-bird-core-3.0.7.jar | grep
>>>> ProtobufConverter
>>>>   4825 Mon Mar 04 16:50:46 UTC 2013
>>>> com/twitter/elephantbird/mapreduce/io/ProtobufConverter.class
>>>>    732 Mon Mar 04 16:50:46 UTC 2013
>>>> com/twitter/elephantbird/mapreduce/io/ProtobufConverter$1.class
>>>>
>>>>
>>>> ------------------------------
>>>> From: vkavuluri@outlook.com
>>>> To: user@hive.apache.org
>>>> Subject: RE: difference between add jar in hive session and hive
>>>> --auxpath
>>>> Date: Thu, 7 Mar 2013 16:44:41 -0800
>>>>
>>>>
>>>> If properly done, "add jar <jar-file>" should work the same as passing
>>>> the jar with --auxpath. Can you run "list jars;" command from CLI or Hue
>>>> and check if you see the jar file.
>>>>
>>>> ------------------------------
>>>> From: java8964@hotmail.com
>>>> To: user@hive.apache.org
>>>> Subject: difference between add jar in hive session and hive --auxpath
>>>> Date: Thu, 7 Mar 2013 17:47:26 -0500
>>>>
>>>>  Hi,
>>>>
>>>> I have a hive table which uses the jar file provided from the
>>>> elephant-bird, which is a framework integrated between lzo and google
>>>> protobuf data and hadoop/hive.
>>>>
>>>> If I use the hive command like this:
>>>>
>>>> hive --auxpath path_to_jars, it works fine to query my table,
>>>>
>>>> but if I use the add jar after I started the hive session, I will get
>>>> ClassNotFoundException in the runtime of my query of the classes in those
>>>> jars.
>>>>
>>>> My questions are:
>>>>
>>>> 1) What is the different between hive --auxpath and "add jar" in the
>>>> hive session?
>>>> 2) This problem makes it is hard to access my table in the HUE, as it
>>>> only supports "add jar", but not --auxpath option. Any suggestions?
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Yong
>>>>
>>>
>>>
>>>
>>> --
>>> *Dean Wampler, Ph.D.*
>>> thinkbiganalytics.com
>>> +1-312-339-1330
>>>
>>>
>>
>

Mime
View raw message