hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Maritato <fmarit...@attinteractive.com>
Subject Re: Building dependencies into UDAF jar
Date Mon, 07 Nov 2011 19:38:37 GMT
I'm not sure how to do this in ant, but it definitely works with maven+assembly because this
is what we do for our production map reduce jobs and for our hive udf's. If you want I can
probably dig up an example pom.xml that shows how to do this.
--
Frank Maritato

From: Pradeep Kamath <pradeepkth@yahoo.com<mailto:pradeepkth@yahoo.com>>
Reply-To: <user@hive.apache.org<mailto:user@hive.apache.org>>, Pradeep Kamath
<pradeepkth@yahoo.com<mailto:pradeepkth@yahoo.com>>
Date: Mon, 7 Nov 2011 11:25:15 -0800
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: Building dependencies into UDAF jar

Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with
maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly"
created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating
a jar which hive/hadoop will understand to be a "composite" jar containing main classes and
other dependent jars?

________________________________
From: Frank Maritato <fmaritato@attinteractive.com<mailto:fmaritato@attinteractive.com>>
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>;
Pradeep Kamath <pradeepkth@yahoo.com<mailto:pradeepkth@yahoo.com>>
Sent: Monday, November 7, 2011 10:47 AM
Subject: Re: Building dependencies into UDAF jar

Hi Pradeep,

If you are using maven to build your code you can use the assembly plugin to produce a jar
with dependencies.
http://maven.apache.org/plugins/maven-assembly-plugin/

Hope this helps
--
Frank Maritato

From: Pradeep Kamath <pradeepkth@yahoo.com<mailto:pradeepkth@yahoo.com>>
Reply-To: <user@hive.apache.org<mailto:user@hive.apache.org>>, Pradeep Kamath
<pradeepkth@yahoo.com<mailto:pradeepkth@yahoo.com>>
Date: Mon, 7 Nov 2011 10:21:26 -0800
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Building dependencies into UDAF jar

Hi,
  I am trying to build a hive UDAF which has dependencies on other external jars (like log4j)
- is there a way in which I can bundle the main classes for my UDAF and all the dependent
jars into one jar for use in my hive query. I tried having my UDF classes at the top level
and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar"
but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH
environment variable does work but requires that I have all the dependent jars separately
on the local filesystem where I run the hive query from. It would be more convenient for me
to build one self contained jar and refer to that in my query (due to some restrictions in
the cluster I run the query in) - is that possible?

Thanks,
Pradeep



Mime
View raw message