hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kawa <kawa.a...@gmail.com>
Subject Re: How to specify Hive auxiliary jar in HDFS, not local file system
Date Mon, 02 Dec 2013 22:37:37 GMT
You can use ADD JAR command inside a Hive script and a parameter in Oozie
workflow definition. Example is here:
http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/


2013/12/2 <mpeterson2@gmail.com>

> Is it possible to specify a Hive auxiliary jar (like a SerDe) that is in
> HDFS rather than the local fileystem?
>
> I am using a CsvSerDe I wrote and when I specify it Hive
> hive.aux.jars.path with a local file system path it works:
>
> hive -hiveconf hive.aux.jars.path=*file:*///path/to/truven-hive-serdes-1.0.jar
> -hiveconf hive.auto.convert.join.noconditionaltask.size=25000000 -f
> hivefiscalyearqueries.sql
>
> But when I put that jar in HDFS and point it to, it fails:
>
> hive -hiveconf hive.aux.jars.path=*hdfs:*///hdfspath/to/truven-hive-serdes-1.0.jar
> -hiveconf hive.auto.convert.join.noconditionaltask.size=25000000 -f
> hivefiscalyearqueries.sql
>
> with the error message:
>
> java.lang.ClassNotFoundException: com.truven.hiveserde.csv.CsvSerDe
> Continuing ...
> 2013-12-02 03:48:25     Starting to launch local task to process map join;
>      maximum memory = 1065484288
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception
> nulljava.lang.NullPointerException
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRowInspectorFromTable(FetchOperator.java:230)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getOutputObjectInspector(FetchOperator.java:595)
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:406)
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:290)
>         at
> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:682)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
>
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getOutputObjectInspector(FetchOperator.java:631)
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:406)
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:290)
>         at
> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:682)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> Execution failed with exit status: 2
> Obtaining error information
>
> Task failed!
>
>
>
> I will need to run this from Oozie eventually, so I'd like to know how get
> Hive to use a jar in HDFS, rather than have to distribute the file to the
> local file system of all datanodes.
>
> Thank you,
> Michael
>

Mime
View raw message