spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Herman van Hovell tot Westerflier (JIRA)" <>
Subject [jira] [Commented] (SPARK-4233) Simplify the Aggregation Function implementation
Date Fri, 22 May 2015 18:39:17 GMT


Herman van Hovell tot Westerflier commented on SPARK-4233:


I have looked through the code in the PR. The new interface doesn't look simpler to me. It
seems that it has been design with Hive UDAFs in mind.

Can you explain to me why the current UDAF implementation is complicated, why it needs to
change, and what is improved if we start to use the proposed implementation?

As for the distinct implementations. Why not nest the required aggregation operator in a distinct
operator? For instance:
case class DistinctifyFunction(
    @transient expr: Seq[Expression],
    @transient aggr: AggregateFunction
    @transient base: AggregateExpression)
  extends AggregateFunction {

  def this() = this(null, null) // Required for serialization.

  val seen = new OpenHashSet[Any]()

  val distinctValue = new InterpretedProjection(expr)

  override def update(input: Row): Unit = {
    val evaluatedExpr = distinctValue(input)
    if (!evaluatedExpr.anyNull) {

  override def eval(input: Row): Any = {
    // Assume the AggregateFunction input has been rerouted, to the distinct value projection.

> Simplify the Aggregation Function implementation
> ------------------------------------------------
>                 Key: SPARK-4233
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Cheng Hao
> Currently, the UDAF implementation is quite complicated, and we have to provide distinct
& non-distinct version.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message