hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Problem with UDF
Date Thu, 30 Apr 2009 01:03:30 GMT
Hey,
I made my own UDF, but I am having trouble getting it running . Any hints?
Thanks

[hadoop@hadoop1 ~]$ /opt/hive/bin/hive -hiveconf hive.root.logger=INFO,console

Hive history file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
09/04/29 20:45:28 INFO exec.HiveHistory: Hive history
file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
hive> create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO';
09/04/29 20:45:34 INFO parse.ParseDriver: Parsing command:
create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
09/04/29 20:45:34 INFO parse.ParseDriver: Parse Completed
09/04/29 20:45:34 INFO parse.FunctionSemanticAnalyzer: analyze done
09/04/29 20:45:34 INFO ql.Driver: Semantic Analysis Completed
09/04/29 20:45:34 INFO ql.Driver: Starting command:
create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
geoip
OK
09/04/29 20:45:34 INFO ql.Driver: OK
Time taken: 0.421 seconds
09/04/29 20:45:34 INFO CliDriver: Time taken: 0.421 seconds

09/04/29 20:46:02 INFO CliDriver: Time taken: 0.06 seconds
hive> select UDFGEO('209.191.139.200') from bigdata limit  1 ;
09/04/29 20:46:09 INFO parse.ParseDriver: Parsing command:
select UDFGEO('209.191.139.200') from bigdata limit  1
09/04/29 20:46:09 INFO parse.ParseDriver: Parse Completed
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Completed phase 1 of
Semantic Analysis
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/04/29 20:46:09 INFO metastore.HiveMetaStore: 0: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/04/29 20:46:09 INFO metastore.ObjectStore: ObjectStore, initialize called
09/04/29 20:46:09 INFO metastore.ObjectStore: found resource
jpox.properties at file:/opt/hive/conf/jpox.properties
09/04/29 20:46:12 INFO metastore.ObjectStore: Initialized ObjectStore
09/04/29 20:46:12 INFO metastore.HiveMetaStore: 0: get_table :
db=default tbl=bigdata
09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
double clus, double dst, double enumber, double etime, double hist,
double nlb, double qxb, double rnumber, double tracks, double vertex,
double zdc}
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
destination tables
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed getting
MetaData in Semantic Analysis
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: get_table :
db=default tbl=bigdata
09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/04/29 20:46:13 INFO metastore.ObjectStore: ObjectStore, initialize called
09/04/29 20:46:13 INFO metastore.ObjectStore: Initialized ObjectStore
09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
double clus, double dst, double enumber, double etime, double hist,
double nlb, double qxb, double rnumber, double tracks, double vertex,
double zdc}
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
destination tables
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed partition pruning
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed sample pruning
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed plan generation
09/04/29 20:46:13 INFO ql.Driver: Semantic Analysis Completed
09/04/29 20:46:13 INFO ql.Driver: Starting command:
select UDFGEO('209.191.139.200') from bigdata limit  1
Total MapReduce jobs = 1
09/04/29 20:46:13 INFO ql.Driver: Total MapReduce jobs = 1
Number of reduce tasks is set to 0 since there's no reduce operator
09/04/29 20:46:13 INFO exec.ExecDriver: Number of reduce tasks is set
to 0 since there's no reduce operator
09/04/29 20:46:13 INFO exec.ExecDriver: Adding input file
/user/hive/warehouse/bigdata
09/04/29 20:46:13 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
09/04/29 20:46:13 INFO mapred.FileInputFormat: Total input paths to process : 1
Starting Job = job_200904292034_0004, Tracking URL =
http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
09/04/29 20:46:14 INFO exec.ExecDriver: Starting Job =
job_200904292034_0004, Tracking URL =
http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
Kill Command = /opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
-Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
job_200904292034_0004
09/04/29 20:46:14 INFO exec.ExecDriver: Kill Command =
/opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
-Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
job_200904292034_0004
 map = 0%,  reduce =0%
09/04/29 20:46:15 INFO exec.ExecDriver:  map = 0%,  reduce =0%
 map = 100%,  reduce =100%
09/04/29 20:46:43 INFO exec.ExecDriver:  map = 100%,  reduce =100%
Ended Job = job_200904292034_0004 with errors
09/04/29 20:46:43 ERROR exec.ExecDriver: Ended Job =
job_200904292034_0004 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver
09/04/29 20:46:43 ERROR ql.Driver: FAILED: Execution Error, return
code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
Time taken: 34.021 seconds
09/04/29 20:46:43 INFO CliDriver: Time taken: 34.021 seconds
hive>



package com.xxx.sysops.udf;

import com.maxmind.geoip.LookupService;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

/**
 *
 * @author ecapriolo
 */
public class UDFGEO extends UDF{
 static LookupService ls;
    static {
      try {
      ls= new LookupService("/usr/local/share/GeoIP/GeoIPCity.dat",
					LookupService.GEOIP_MEMORY_CACHE );
      } catch (java.io.IOException ex) { System.err.println("geoip");}
    }
    Text result= new Text();
    public UDFGEO(){

    }
    public Text evaluate(Text t){
      if (t == null)
        return null;
      try{
        //result.set(ls.getLocation(t.toString()).region);
        result.set("blabla");
      } catch (Exception ex){System.err.println(ex);}
      return  result;
    }
}

Mime
View raw message