Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 932827384 for ; Mon, 7 Nov 2011 19:46:19 +0000 (UTC) Received: (qmail 77399 invoked by uid 500); 7 Nov 2011 19:46:18 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 77361 invoked by uid 500); 7 Nov 2011 19:46:18 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 77353 invoked by uid 99); 7 Nov 2011 19:46:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Nov 2011 19:46:18 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.220.48] (HELO mail-dy0-f48.google.com) (209.85.220.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Nov 2011 19:46:11 +0000 Received: by dyj10 with SMTP id 10so135849dyj.35 for ; Mon, 07 Nov 2011 11:45:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.2.164 with SMTP id 4mr9511199obv.49.1320695149796; Mon, 07 Nov 2011 11:45:49 -0800 (PST) Received: by 10.182.225.101 with HTTP; Mon, 7 Nov 2011 11:45:49 -0800 (PST) X-Originating-IP: [69.8.212.254] In-Reply-To: References: <1320693915.78403.YahooMailNeo@web120529.mail.ne1.yahoo.com> Date: Mon, 7 Nov 2011 13:45:49 -0600 Message-ID: Subject: Re: Building dependencies into UDAF jar From: Mark Kerzner To: user@hive.apache.org Cc: Pradeep Kamath Content-Type: multipart/alternative; boundary=f46d044519e1429e0304b12a4dd9 X-Virus-Checked: Checked by ClamAV on apache.org --f46d044519e1429e0304b12a4dd9 Content-Type: text/plain; charset=ISO-8859-1 Here is an example of assembly plugin, https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml Mark On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato wrote: > I'm not sure how to do this in ant, but it definitely works with > maven+assembly because this is what we do for our production map reduce > jobs and for our hive udf's. If you want I can probably dig up an example > pom.xml that shows how to do this. > -- > Frank Maritato > > From: Pradeep Kamath > Reply-To: , Pradeep Kamath > Date: Mon, 7 Nov 2011 11:25:15 -0800 > To: "user@hive.apache.org" > Subject: Re: Building dependencies into UDAF jar > > Thanks for the suggestion Frank. Unfortunately I am using ant and am > not very familiar with maven. Glancing through the link on > maven-assembly-plugin, I am also not sure if the "assembly" created will > work fine with hive/hadoop when the task runs on a remote node. Any way of > creating a jar which hive/hadoop will understand to be a "composite" jar > containing main classes and other dependent jars? > > ------------------------------ > *From:* Frank Maritato > *To:* "user@hive.apache.org" ; Pradeep Kamath < > pradeepkth@yahoo.com> > *Sent:* Monday, November 7, 2011 10:47 AM > *Subject:* Re: Building dependencies into UDAF jar > > Hi Pradeep, > > If you are using maven to build your code you can use the assembly > plugin to produce a jar with dependencies. > http://maven.apache.org/plugins/maven-assembly-plugin/ > > Hope this helps > -- > Frank Maritato > > From: Pradeep Kamath > Reply-To: , Pradeep Kamath > Date: Mon, 7 Nov 2011 10:21:26 -0800 > To: "user@hive.apache.org" > Subject: Building dependencies into UDAF jar > > Hi, > I am trying to build a hive UDAF which has dependencies on other > external jars (like log4j) - is there a way in which I can bundle the main > classes for my UDAF and all the dependent jars into one jar for use in my > hive query. I tried having my UDF classes at the top level and dependency > jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" > but that did not work (classes from dependency jars were not being > resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but > requires that I have all the dependent jars separately on the local > filesystem where I run the hive query from. It would be more convenient for > me to build one self contained jar and refer to that in my query (due to > some restrictions in the cluster I run the query in) - is that possible? > > Thanks, > Pradeep > > > --f46d044519e1429e0304b12a4dd9 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Here is an example of assembly plugin, https://github.com/markkerz= ner/HadoopInPracticeCode/blob/master/pom.xml

Mark

On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <fmaritato@attinteractive.com&= gt; wrote:
I'm not sure how to do this in ant, but it definitely works with m= aven+assembly because this is what we do for our production map reduce jobs= and for our hive udf's. If you want I can probably dig up an example p= om.xml that shows how to do this.
--=A0
Frank Maritato

From: Pradeep Kamath <pradeepkth@yahoo.com>= ;
Reply-To: <user@hive.apache.org>, Pradeep = Kamath <pradee= pkth@yahoo.com>
Date: Mon, 7 Nov 2011 11:25:15 -080= 0
To: "user@hive.apache.org" <user@hive.apache.org= >
Subject: Re: Building dependencies = into UDAF jar

Thanks for the suggestion Frank. Unfortunately I am using ant an= d am not very familiar with maven. Glancing through the link on maven-assem= bly-plugin, I am also not sure if the "assembly" created will wor= k fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will underst= and to be a "composite" jar containing main classes and other dep= endent jars?

From: Frank Maritato <fmaritato@at= tinteractive.com>
To: "user@hive.apache.org" <= user@hive.apache.= org>; Pradeep Kamath <pradeepkth@yahoo.com>
Sent: Monday, November 7, 20= 11 10:47 AM
Subject: Re: Building depend= encies into UDAF jar

Hi Pradeep,

If you are using maven to build your code you can use the assembly plu= gin to produce a jar with dependencies.=A0

Hope this helps
--=A0
Frank Maritato

From: Pradeep Kamath <pradeepkth= @yahoo.com>
Reply-To: <user@hive.apache.org<= /a>>, Pradeep Kamath <pradeepkth@yahoo.com>
Date: Mon, 7 Nov 2011 10:21:26 -080= 0
To: "user@hive.apache.org= " <user@hive.apache.org>
Subject: Building dependencies into= UDAF jar

Hi,
=A0 I am trying to build a hive UDAF which has dependencies on other e= xternal jars (like log4j) - is there a way in which I can bundle the main c= lasses for my UDAF and all the dependent jars into one jar for use in my hi= ve query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside= one jar and then using "add jar UDAF.jar" but that did not work = (classes from dependency jars were not being resolved). Setting HIVE_AUX_JA= RS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem wher= e I run the hive query from. It would be more convenient for me to build on= e self contained jar and refer to that in my query (due to some restriction= s in the cluster I run the query in) - is that possible?

Thanks,
Pradeep

--f46d044519e1429e0304b12a4dd9--