Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D0DE69D65 for ; Fri, 3 Feb 2012 01:35:01 +0000 (UTC) Received: (qmail 21795 invoked by uid 500); 3 Feb 2012 01:35:00 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 21656 invoked by uid 500); 3 Feb 2012 01:34:59 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 21647 invoked by uid 99); 3 Feb 2012 01:34:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Feb 2012 01:34:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Feb 2012 01:34:52 +0000 Received: by iagz35 with SMTP id z35so5530093iag.35 for ; Thu, 02 Feb 2012 17:34:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=05l3iH4XFmZuUOY/7ksLztxqyOdKACxGb+K/G/3bC+0=; b=iLzJcs5vQ7yP71pDjNRlBzrRB4oFy0q7IX1VTqJW2yOXYhd/KN+6dKX6QAbKeAmIMP iFyXY4C42Rr4E4tC6kX8jjOpPZZpZxZveXUO55rNR4IKBoBt54D27ApX6SLufKqHZJ6t QnowJ2VfCQJrTcjXVhcyBsz19Z8+OKOYRpAv4= MIME-Version: 1.0 Received: by 10.50.189.134 with SMTP id gi6mr15065012igc.18.1328232871607; Thu, 02 Feb 2012 17:34:31 -0800 (PST) Received: by 10.42.197.1 with HTTP; Thu, 2 Feb 2012 17:34:31 -0800 (PST) In-Reply-To: References: Date: Thu, 2 Feb 2012 20:34:31 -0500 Message-ID: Subject: Re: UDF ;; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask From: Edward Capriolo To: user@hive.apache.org Content-Type: multipart/alternative; boundary=14dae9340a1b7dbe4804b8055000 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9340a1b7dbe4804b8055000 Content-Type: text/plain; charset=ISO-8859-1 Those are pretty crappy instructions. Best to build jars with maven or ant or an iDE. On Thu, Feb 2, 2012 at 8:27 PM, Time Less wrote: > I sat down with one of our Java engineers who solved it. It actually > appears we're being affected by a slightly-incorrect set of instructions on > the internet. The typical web instructions (duplicated in many places on > the web, such as here > http://www.findnwrite.com/musings/extract-top-n-records-in-each-group-in-hadoophive/) > say to build the .jar like so: > > *$ jar -cf LeoRank.jar /tmp/jar/com/riot/hive/udf/LeoRank.class > * > However, this creates a JAR like so: > > *$ jar -tf LeoRank.jar > META-INF/ > META-INF/MANIFEST.MF > /tmp/jar/com/riot/hive/udf/LeoRank.class > * > Note the class location in the JAR! But we're creating the package as > com.riot.hive.udf, NOT as tmp.jar.com.riot.hive.udf, so it breaks. The > solution is to change the "jar -cf" command like so: > > *$ cd /tmp/jar > $ jar -cf LeoRank.jar ./com/riot/hive/udf/LeoRank.class > $ jar -tf LeoRank.jar > META-INF/ > META-INF/MANIFEST.MF > com/riot/hive/udf/LeoRank.class > * > Note the now-proper class location in the JAR. > > The error is quite generic (read the wide variety of advices given to > people getting this error, from "set your classpath" to "name your function > correctly"). Since the commonly-referenced web instructions for this seem > to lead people astray, hopefully this post will help others in the future. > > -- > Tim Ellis > Data Architect, Riot Games > > > On Thu, Feb 2, 2012 at 5:08 PM, Edward Capriolo wrote: > >> Maybe you are being effected by this? >> >> https://issues.apache.org/jira/browse/HIVE-2635 >> >> >> >> >> On Thu, Feb 2, 2012 at 6:38 PM, Time Less wrote: >> >>> I'm having a heck of a time the past couple days. Google suggests others >>> have had this same error without resolution since mid-2010. Maybe someone >>> here can shed some light on this? >>> >>> *See the package and method name: >>> *[hdfs@laxhadoop1-012 15:23:14 ~/Tim] :) head LeoRank.java >>> package com.riot.hive.udf; >>> import org.apache.hadoop.hive.ql.exec.UDF; >>> public final class *LeoRank* extends UDF{ >>> >>> *We set the classpath and ensure .jar is there: >>> *[hdfs@laxhadoop1-012 15:24:35 ~/Tim] :) export >>> HADOOP_CLASSPATH=/usr/lib/hive/lib >>> [hdfs@laxhadoop1-012 15:24:44 ~/Tim] :) ls -al >>> $HADOOP_CLASSPATH/LeoRank.jar >>> -rw-rw-r-- 1 hdfs hdfs 884 Feb 2 15:07 /usr/lib/hive/lib/LeoRank.jar >>> >>> *We add the jar: >>> *[hdfs@laxhadoop1-012 15:24:52 ~/Tim] :) hive >>> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201202021524_1137940055.txt >>> hive> add jar LeoRank.jar ; >>> Added LeoRank.jar to class path >>> Added resource: LeoRank.jar >>> >>> *And double-check the .jar is loaded: >>> *hive> list jars ; >>> LeoRank.jar >>> >>> *And yet... fail:* >>> hive> create temporary function LeoRank as 'com.riot.hive.udf.LeoRank' ; >>> FAILED: Execution Error, return code 1 from >>> org.apache.hadoop.hive.ql.exec.FunctionTask >>> >>> *hive.log doesn't seem to shed any additional light on it (or does it?): >>> * >>> [hdfs@laxhadoop1-012 15:31:16 ~/Tim] :) tail -f /tmp/hdfs/hive.log >>> 2012-02-02 15:31:13,087 ERROR ql.Driver >>> (SessionState.java:printError(365)) - FAILED: Execution Error, return code >>> 1 from org.apache.hadoop.hive.ql.exec.FunctionTask >>> >>> There are posts on forums through history where people go through >>> exactly this process, get to exactly this point, then nothing. I'm not sure >>> if they give up on UDFs, or if they have some brain flash around a >>> configuration they forgot to set or what. Any clues for the gamers of the >>> world? >>> >>> -- >>> Tim Ellis >>> Data Architect, Riot Games >>> >>> >> > --14dae9340a1b7dbe4804b8055000 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Those are pretty crappy instructions. Best to build jars with maven or ant = or an iDE.

On Thu, Feb 2, 2012 at 8:27 PM= , Time Less <timelessness@gmail.com> wrote:
I sat down with one of ou= r Java engineers who solved it. It actually appears we're being affecte= d by a slightly-incorrect set of instructions on the internet. The typical = web instructions (duplicated in many places on the web, such as here http://www.findnwrite.com/musings/extract= -top-n-records-in-each-group-in-hadoophive/) say to build the .jar like= so:

$ jar -cf LeoRank.jar /tmp/jar/com/riot/hive/udf/LeoRank.class

However, this creates a JAR like so:

$ jar -tf LeoRank.jar<= br>META-INF/
META-INF/MANIFEST.MF
/tmp/jar/com/riot/hive/udf/LeoRank.= class

Note the class location in the JAR! But we're creating the pack= age as com.riot.hive.udf, NOT as tmp.jar.com.riot.hive.udf, so it breaks. T= he solution is to change the "jar -cf" command like so:

$ cd /tmp/jar
$ jar -cf LeoRank.jar ./com/riot/hive/udf/LeoRank.class=
$ jar -tf LeoRank.jar
META-INF/
META-INF/MANIFEST.MF
com/riot/= hive/udf/LeoRank.class

Note the now-proper class location in the= JAR.

The error is quite generic (read the wide variety of advices given to p= eople getting this error, from "set your classpath" to "name= your function correctly"). Since the commonly-referenced web instruct= ions for this seem to lead people astray, hopefully this post will help oth= ers in the future.

--
Tim Ellis
Data Architect, Riot Games


On Thu, Feb 2, 2012 at 5:08 PM, Edward Capriolo <edlinuxg= uru@gmail.com> wrote:
Maybe you are being effec= ted by this?

https://issues.apache.org/jira/browse/HIVE-2635<= div>




On Thu, Feb 2, 2012 at 6:38= PM, Time Less <timelessness@gmail.com> wrote:
I'm having a heck of = a time the past couple days. Google suggests others have had this same erro= r without resolution since mid-2010. Maybe someone here can shed some light= on this?

See the package and method name:
[hdfs@laxhadoop1-012 15:23:14 ~/Tim] :) head LeoRank.java
package c= om.riot.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
public f= inal class LeoRank extends UDF{

We set the classpath and e= nsure .jar is there:
[hdfs@laxhadoop1-012 15:24:35 ~/Tim] :) export HADOOP_CLASSPATH=3D/usr/= lib/hive/lib
[hdfs@laxhadoop1-012 15:24:44 ~/Tim] :) ls -al $HADOOP_CLAS= SPATH/LeoRank.jar
-rw-rw-r-- 1 hdfs hdfs 884 Feb=A0 2 15:07 /usr/lib/hiv= e/lib/LeoRank.jar

We add the jar:
[hdfs@laxhadoop1-012 15:24:52 ~/Tim] :) hive<= br>Hive history file=3D/tmp/hdfs/hive_job_log_hdfs_201202021524_1137940055.= txt
hive> add jar LeoRank.jar ;
Added LeoRank.jar to class path Added resource: LeoRank.jar

And double-check the .jar is loaded:<= br>hive> list jars ;
LeoRank.jar

And yet... fail:hive> create temporary function LeoRank as 'com.riot.hive.udf.LeoR= ank' ;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.= FunctionTask

hive.log doesn't seem to shed any additional lig= ht on it (or does it?):
[hdfs@laxhadoop1-012 15:31:16 ~/Tim] :) tail= -f /tmp/hdfs/hive.log
2012-02-02 15:31:13,087 ERROR ql.Driver (SessionState.java:printError(365))= - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ex= ec.FunctionTask

There are posts on forums through history where peop= le go through exactly this process, get to exactly this point, then nothing= . I'm not sure if they give up on UDFs, or if they have some brain flas= h around a configuration they forgot to set or what. Any clues for the game= rs of the world?

--
Tim Ellis
Data Architect, Riot Games




--14dae9340a1b7dbe4804b8055000--