hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surbhit (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-12719) As a hive user, I am facing issues using permanent UDAF's.
Date Mon, 21 Dec 2015 10:54:46 GMT

     [ https://issues.apache.org/jira/browse/HIVE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Surbhit updated HIVE-12719:
---------------------------
    Release Note: 
I am facing an issues while calling custom generic UDAF's in HIVE (1.2.1).

Below is my implementation for GenericUDAF : - 

/**
 * 
 */
package com.practice.hive.udaf.test;

import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.parse.SemanticException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFParameterInfo;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
import org.apache.hadoop.hive.ql.util.JavaDataModel;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.LongObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.io.LongWritable;

/**
 * @author surbhit.shrivastava
 *
 */
public class GenericUDAFRegrCount implements GenericUDAFResolver2 {

//	 This is deprecated will not be called .. but is present for backward compatibility purpose
	@Override
	public GenericUDAFEvaluator getEvaluator(TypeInfo[] parameters)
			throws SemanticException {

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}
		return new GenericUDAFRegrCountEvaluator();
	}

	@Override
	public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo paramInfo)
			throws SemanticException {

		TypeInfo[] parameters = paramInfo.getParameters();

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}

		if (paramInfo.isAllColumns()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"* is not supported");
		}

		if (paramInfo.isDistinct()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"DISTINCT keyword is not allowed");
		}
		return new GenericUDAFRegrCountEvaluator();

	}

	public static class GenericUDAFRegrCountEvaluator extends
			GenericUDAFEvaluator {

		private LongObjectInspector partialCountAggOI;
		private LongWritable result;

		@Override
		public ObjectInspector init(Mode m, ObjectInspector[] parameters)
				throws HiveException {
			super.init(m, parameters);
			if (m == Mode.PARTIAL2 || m == Mode.FINAL) {
				partialCountAggOI = (LongObjectInspector) parameters[0];
			}
			result = new LongWritable(0);
			return PrimitiveObjectInspectorFactory.writableLongObjectInspector;
		}

		/** class for storing count value. */
		@AggregationType(estimable = true)
		static class CountAgg extends AbstractAggregationBuffer {
			long value;

			@Override
			public int estimate() {
				return JavaDataModel.PRIMITIVES2;
			}
		}

		@Override
		public AggregationBuffer getNewAggregationBuffer() throws HiveException {
			CountAgg buffer = new CountAgg();
			reset(buffer);
			return buffer;
		}

		@Override
		public void reset(AggregationBuffer agg) throws HiveException {
			((CountAgg) agg).value = 0;
		}

		@Override
		public void iterate(AggregationBuffer agg, Object[] parameters)
				throws HiveException {

			if (parameters == null) {
				return;
			}

			if (null != parameters[0] && null != parameters[1]) {
				((CountAgg) agg).value++;
			}
		}

		@Override
		public void merge(AggregationBuffer agg, Object partial) throws HiveException {
			if (partial != null) {
				long p = partialCountAggOI.get(partial);
				((CountAgg) agg).value += p;
			}
		}

		@Override
		public Object terminate(AggregationBuffer agg) throws HiveException {
			result.set(((CountAgg) agg).value);
			return result;
		}

		@Override
		public Object terminatePartial(AggregationBuffer agg)
				throws HiveException {
			return terminate(agg);
		}
	}
}

I have registered the function using -

CREATE FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount' USING JAR
'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

I get the below errors while trying to execute query -

select regr_count(empid,depid) over() from EMPLOYEE;

FAILED: SemanticException Failed to breakup Windowing invocations into Groups. At least 1
group must only depend on input columns. Also check for circular dependencies. Underlying
error: Invalid function regr_count

Although the same function works fine when registered as temporary function using .

CREATE TEMPORARY FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount'
USING JAR 'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

Any help would be highly appreciated :)

  was:
I am facing an issues while calling custom generic UDAF's in HIVE (1.2.1).

Below is my implementation for GenericUDAF : - 
{code}
/**
 * 
 */
package com.practice.hive.udaf.test;

import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.parse.SemanticException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFParameterInfo;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
import org.apache.hadoop.hive.ql.util.JavaDataModel;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.LongObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.io.LongWritable;

/**
 * @author surbhit.shrivastava
 *
 */
public class GenericUDAFRegrCount implements GenericUDAFResolver2 {

//	 This is deprecated will not be called .. but is present for backward compatibility purpose
	@Override
	public GenericUDAFEvaluator getEvaluator(TypeInfo[] parameters)
			throws SemanticException {

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}
		return new GenericUDAFRegrCountEvaluator();
	}

	@Override
	public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo paramInfo)
			throws SemanticException {

		TypeInfo[] parameters = paramInfo.getParameters();

		if (parameters.length != 2) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"Exactly two arguments are expected.");
		}

		if (paramInfo.isAllColumns()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"* is not supported");
		}

		if (paramInfo.isDistinct()) {
			throw new UDFArgumentTypeException(parameters.length - 1,
					"DISTINCT keyword is not allowed");
		}
		return new GenericUDAFRegrCountEvaluator();

	}

	public static class GenericUDAFRegrCountEvaluator extends
			GenericUDAFEvaluator {

		private LongObjectInspector partialCountAggOI;
		private LongWritable result;

		@Override
		public ObjectInspector init(Mode m, ObjectInspector[] parameters)
				throws HiveException {
			super.init(m, parameters);
			if (m == Mode.PARTIAL2 || m == Mode.FINAL) {
				partialCountAggOI = (LongObjectInspector) parameters[0];
			}
			result = new LongWritable(0);
			return PrimitiveObjectInspectorFactory.writableLongObjectInspector;
		}

		/** class for storing count value. */
		@AggregationType(estimable = true)
		static class CountAgg extends AbstractAggregationBuffer {
			long value;

			@Override
			public int estimate() {
				return JavaDataModel.PRIMITIVES2;
			}
		}

		@Override
		public AggregationBuffer getNewAggregationBuffer() throws HiveException {
			CountAgg buffer = new CountAgg();
			reset(buffer);
			return buffer;
		}

		@Override
		public void reset(AggregationBuffer agg) throws HiveException {
			((CountAgg) agg).value = 0;
		}

		@Override
		public void iterate(AggregationBuffer agg, Object[] parameters)
				throws HiveException {

			if (parameters == null) {
				return;
			}

			if (null != parameters[0] && null != parameters[1]) {
				((CountAgg) agg).value++;
			}
		}

		@Override
		public void merge(AggregationBuffer agg, Object partial) throws HiveException {
			if (partial != null) {
				long p = partialCountAggOI.get(partial);
				((CountAgg) agg).value += p;
			}
		}

		@Override
		public Object terminate(AggregationBuffer agg) throws HiveException {
			result.set(((CountAgg) agg).value);
			return result;
		}

		@Override
		public Object terminatePartial(AggregationBuffer agg)
				throws HiveException {
			return terminate(agg);
		}
	}
}

{code}

I have registered the function using -

CREATE FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount' USING JAR
'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

I get the below errors while trying to execute query -

select regr_count(empid,depid) over() from EMPLOYEE;

FAILED: SemanticException Failed to breakup Windowing invocations into Groups. At least 1
group must only depend on input columns. Also check for circular dependencies. Underlying
error: Invalid function regr_count

Although the same function works fine when registered as temporary function using .

CREATE TEMPORARY FUNCTION REGR_COUNT AS 'com.practice.hive.udaf.test.GenericUDAFRegrCount'
USING JAR 'hdfs:///data/test/hadoop-sample-0.1-SNAPSHOT.jar';

Any help would be highly appreciated :)


> As a hive user, I am facing issues using permanent UDAF's.
> ----------------------------------------------------------
>
>                 Key: HIVE-12719
>                 URL: https://issues.apache.org/jira/browse/HIVE-12719
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Surbhit
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message