hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesha Shreedhara (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-12719) As a hive user, I am facing issues using permanent UDAF's.
Date Mon, 11 Dec 2017 14:52:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ganesha Shreedhara updated HIVE-12719:
--------------------------------------
    Release Note: 
We will get invalid function error when permanent function is used along with window function
because the permanent function is stored as db_name.function_name in the function registry
where as built-in/temporary function gets stored without any prefix in the function registry.

The fix is done such that the we will first form the qualified name for the permanent function
and then compare if it exists in function registry. 

  was:
I am facing an issues while calling custom generic UDAF's in HIVE (1.2.1).

Below is my implementation for GenericUDAF : - 

/**
 * 
 */
package com.practice.hive.udaf.test;

import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.parse.SemanticException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFParameterInfo;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
import org.apache.hadoop.hive.ql.util.JavaDataModel;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.LongObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.io.LongWritable;

/**
 * @author surbhit.shrivastava
 *
 */
public class GenericUDAFRegrCount implements GenericUDAFResolver2 {

//	 This is deprecated will not be called .. but is present for backward compatibility purpose
	@Override
	public GenericUDAFEvaluator getEvaluator(TypeInfo[] parameters)
			throws SemanticException {

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}
		return new GenericUDAFRegrCountEvaluator();
	}

	@Override
	public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo paramInfo)
			throws SemanticException {

		TypeInfo[] parameters = paramInfo.getParameters();

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}

		if (paramInfo.isAllColumns()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"* is not supported");
		}

		if (paramInfo.isDistinct()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"DISTINCT keyword is not allowed");
		}
		return new GenericUDAFRegrCountEvaluator();

	}

	public static class GenericUDAFRegrCountEvaluator extends
			GenericUDAFEvaluator {

		private LongObjectInspector partialCountAggOI;
		private LongWritable result;

		@Override
		public ObjectInspector init(Mode m, ObjectInspector[] parameters)
				throws HiveException {
			super.init(m, parameters);
			if (m == Mode.PARTIAL2 || m == Mode.FINAL) {
				partialCountAggOI = (LongObjectInspector) parameters[0];
			}
			result = new LongWritable(0);
			return PrimitiveObjectInspectorFactory.writableLongObjectInspector;
		}

		/** class for storing count value. */
		@AggregationType(estimable = true)
		static class CountAgg extends AbstractAggregationBuffer {
			long value;

			@Override
			public int estimate() {
				return JavaDataModel.PRIMITIVES2;
			}
		}

		@Override
		public AggregationBuffer getNewAggregationBuffer() throws HiveException {
			CountAgg buffer = new CountAgg();
			reset(buffer);
			return buffer;
		}

		@Override
		public void reset(AggregationBuffer agg) throws HiveException {
			((CountAgg) agg).value = 0;
		}

		@Override
		public void iterate(AggregationBuffer agg, Object[] parameters)
				throws HiveException {

			if (parameters == null) {
				return;
			}

			if (null != parameters[0] && null != parameters[1]) {
				((CountAgg) agg).value++;
			}
		}

		@Override
		public void merge(AggregationBuffer agg, Object partial) throws HiveException {
			if (partial != null) {
				long p = partialCountAggOI.get(partial);
				((CountAgg) agg).value += p;
			}
		}

		@Override
		public Object terminate(AggregationBuffer agg) throws HiveException {
			result.set(((CountAgg) agg).value);
			return result;
		}

		@Override
		public Object terminatePartial(AggregationBuffer agg)
				throws HiveException {
			return terminate(agg);
		}
	}
}

I have registered the function using -

CREATE FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount' USING JAR
'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

I get the below errors while trying to execute query -

select regr_count(empid,depid) over() from EMPLOYEE;

FAILED: SemanticException Failed to breakup Windowing invocations into Groups. At least 1
group must only depend on input columns. Also check for circular dependencies. Underlying
error: Invalid function regr_count

Although the same function works fine when registered as temporary function using .

CREATE TEMPORARY FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount'
USING JAR 'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

Any help would be highly appreciated :)

          Status: Patch Available  (was: Open)

> As a hive user, I am facing issues using permanent UDAF's.
> ----------------------------------------------------------
>
>                 Key: HIVE-12719
>                 URL: https://issues.apache.org/jira/browse/HIVE-12719
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Surbhit
>            Assignee: Ganesha Shreedhara
>         Attachments: HIVE-12719.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message