hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "neelesh gadhia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2361) Add some UDFs which help to migrate Oracle to Hive
Date Thu, 14 Feb 2013 07:52:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578207#comment-13578207
] 

neelesh gadhia commented on HIVE-2361:
--------------------------------------

New to using UDF's on hive. But implemented these on hive 0.9.0 and hadoop 1.1.1. Was able
to add jar to hive and kicked the sample sql
SELECT t.empno, t.deptno, t.sal, nexr_sum(hash(t.deptno),t.sal) as sal_sum
FROM (
select a.empno, a.deptno, a.sal from emp a
distribute by hash(a.deptno)
sort BY a.deptno, a.empno
) t;

The sql failed with errors 


2013-02-13 23:30:18,925 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201302132324_0002_r_000000_3'
to tip task_201302132324_0002_r_000000, for tracker 'tracker_192.168.0.151:localhost/127.0.0.1:50099'
2013-02-13 23:30:18,925 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201302132324_0002_r_000000_2'
2013-02-13 23:30:26,484 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201302132324_0002_r_000000_3:
java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:486)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
	... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)
	... 14 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:137)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:922)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
	at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
	... 14 more

2013-02-13 23:30:29,819 INFO org.apache.hadoop.mapred.TaskInProgress: TaskInProgress task_201302132324_0002_r_000000
has failed 4 times.
2013-02-13 23:30:29,820 INFO org.apache.hadoop.mapred.JobInProgress: TaskTracker at '192.168.0.151'
turned 'flaky'
.... 12 more lines..


Tried different function "GenericUDFMax".. same error.

Any pointers/advise, what could be wrong?
                
>  Add some UDFs which help to migrate Oracle to Hive
> ---------------------------------------------------
>
>                 Key: HIVE-2361
>                 URL: https://issues.apache.org/jira/browse/HIVE-2361
>             Project: Hive
>          Issue Type: New Feature
>          Components: UDF
>    Affects Versions: 0.8.0
>            Reporter: JunHo Cho
>            Assignee: JunHo Cho
>            Priority: Minor
>              Labels: features
>         Attachments: nexr-udf.tar
>
>
> Here some UDFs which can be matched to oracle functions:
> There are two kinds of oracle functions. one is scalar function and another is analytic
function.
> Most scalar functions in Oracle can be converted to hive's udf directly.  
> Oracle Scalar Function
> GenericUDFDecode : Compares first argument to each other value one by one. e.g., DECODE(x,0,'zero',1,'one')
will return 'zero' if x is 0
> GenericUDFGreatest : Return the greatest of the list of one or more expressions. e.g.,
GREATEST(2,5,12,3) will return 12
> GenericUDFInstr : Return the location of a substring in a string. e.g., INSTR('next',
'e') will return 2
> GenericUDFLnnvl : Evaluate a condition when one or both operands of the condition may
be null. e.g., LNNVL(2 > 4) will return true
> GenericUDFNVL : Replace null with a string in the results of a query. e.g., NVL(null,'hive')
will return hive
> GenericUDFNVL2 : Determine the value returned by a query based on whether a specified
expression is null or not null. e.g., NVL2(null,'not null','null value') will return 'null
value'
> GenericUDFToNumber : Convert a string to a number. e.g., TO_NUMBER('112','999') will
return 112
> GenericUDFTrunc : Returns a date truncated to a specific unit of measure. e.g., TRUNC('2002-11-02
01:01:01','YYYY') will return '2002-01-01 00:00:00'
> Oracle Analytic Function
> Most analytic functions in Oracle can't be converted to hive's query and udf directly.
> Following udfs should be used with DISTRIBUTED, SORT BY and HASH of hive to support analytic
functions 
> e.q., SELECT _FUNC_(hash(col1), col2, ...) FROM SELECT ~ FROM table DISTRIBUTED BY hash(col1)
SORT BY col1, col2 ...
> GenericUDFSum : Calculate a cumulative sum.
> GenericUDFRank : Assign a sequential order, or rank within some group based on key.
> GenericUDFDenseRank : Act like RANK function except that it assigns consecutive ranks.
> GenericUDFRowNumber : Return sequence integer value within some group based on key.
> GenericUDFMax : Determine the highest value within some group based on key.
> GenericUDFMin : Determine the lowest value within some group based on key.
> GenericUDFLag : Access data from a previous row.
> This udfs was developed with hive-pdk

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message