Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Tue, 1 Mar 2016 11:38:18 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12941314.1456213628000.182278.1456832298069@Atlassian.JIRA>
In-Reply-To: <JIRA.12941314.1456213628000@Atlassian.JIRA>
References: <JIRA.12941314.1456213628000@Atlassian.JIRA>
 <JIRA.12941314.1456213628479@arcas>
Subject: [jira] [Commented] (FLINK-3474) Partial aggregate interface design
 and sort-based implementation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FLINK-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173642#comment-15173642 ] 

ASF GitHub Bot commented on FLINK-3474:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1746#discussion_r54555220
  
    --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/runtime/aggregate/AggregateUtil.scala ---
    @@ -0,0 +1,309 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.flink.api.table.runtime.aggregate
    +
    +import java.util
    +
    +import org.apache.calcite.rel.`type`._
    +import org.apache.calcite.rel.core.AggregateCall
    +import org.apache.calcite.sql.SqlAggFunction
    +import org.apache.calcite.sql.`type`.SqlTypeName._
    +import org.apache.calcite.sql.`type`.{SqlTypeFactoryImpl, SqlTypeName}
    +import org.apache.calcite.sql.fun._
    +import org.apache.flink.api.common.functions.{GroupReduceFunction, MapFunction}
    +import org.apache.flink.api.table.Row
    +import org.apache.flink.api.table.plan.PlanGenException
    +import org.apache.flink.api.table.plan.nodes.logical.FlinkAggregate
    +
    +import scala.collection.JavaConversions._
    +import scala.collection.mutable.ArrayBuffer
    +
    +object AggregateUtil {
    +
    +  /**
    +   * Create Flink operator functions for aggregates. It includes 2 implementations of Flink 
    +   * operator functions:
    +   * [[org.apache.flink.api.common.functions.MapFunction]] and 
    +   * [[org.apache.flink.api.common.functions.GroupReduceFunction]](if it's partial aggregate,
    +   * should also implement [[org.apache.flink.api.common.functions.CombineFunction]] as well). 
    +   * The output of [[org.apache.flink.api.common.functions.MapFunction]] contains the 
    +   * intermediate aggregate values of all aggregate function, it's stored in Row by the following
    +   * format:
    +   *
    +   * {{{
    +   *                   avg(x) aggOffsetInRow = 2          count(z) aggOffsetInRow = 5
    +   *                             |                          |
    +   *                             v                          v
    +   *        +---------+---------+--------+--------+--------+--------+
    +   *        |groupKey1|groupKey2|  sum1  | count1 |  sum2  | count2 |
    +   *        +---------+---------+--------+--------+--------+--------+
    +   *                                              ^
    +   *                                              |
    +   *                               sum(y) aggOffsetInRow = 4
    +   * }}}
    +   *
    +   */
    +  def createOperatorFunctionsForAggregates(aggregate: FlinkAggregate,
    +      inputType: RelDataType, outputType: RelDataType,
    +      groupings: Array[Int]): AggregateResult = {
    +
    +    val aggregateCalls: Seq[AggregateCall] = aggregate.getAggCallList
    +    // store the aggregate fields of each aggregate function, by the same order of aggregates.
    +    val aggFieldIndexes = new Array[Int](aggregateCalls.size)
    +    val aggregates = new Array[Aggregate[_ <: Any]](aggregateCalls.size)
    +
    +    transformToAggregateFunctions(aggregateCalls, aggFieldIndexes,
    +      aggregates, inputType, groupings.length)
    +
    +    val mapFunction = new AggregateMapFunction(aggregates, aggFieldIndexes, groupings)
    +
    +    val bufferDataType: RelRecordType =
    +      createAggregateBufferDataType(groupings, aggregates, inputType)
    +
    +    // the mapping relation between field index of intermediate aggregate Row and output Row.
    +    var groupingOffsetMapping = ArrayBuffer[(Int, Int)]()
    +
    +    // the mapping relation between aggregate function index in list and its corresponding
    +    // field index in output Row.
    +    var aggOffsetMapping = ArrayBuffer[(Int, Int)]()
    +
    +
    +    outputType.getFieldList.zipWithIndex.foreach {
    +      case (fieldType: RelDataTypeField, outputIndex: Int) =>
    +
    +        val aggregateIndex: Int = getMatchedAggregateIndex(aggregate, fieldType)
    +        if (aggregateIndex != -1) {
    +          aggOffsetMapping += ((outputIndex, aggregateIndex))
    +        } else {
    +          val groupKeyIndex: Int = getMatchedFieldIndex(inputType, fieldType, groupings)
    +          if (groupKeyIndex != -1) {
    +            groupingOffsetMapping += ((outputIndex, groupKeyIndex))
    +          } else {
    +            throw new PlanGenException("Could not find output field in input data type " +
    +                "or aggregate function.")
    +          }
    +        }
    +    }
    +
    +    val allPartialAggregate = aggregates.map(_.supportPartial).foldLeft(true)(_ && _)
    --- End diff --
    
    you can do this with `reduce` to avoid the initial `true` element.


> Partial aggregate interface design and sort-based implementation
> ----------------------------------------------------------------
>
>                 Key: FLINK-3474
>                 URL: https://issues.apache.org/jira/browse/FLINK-3474
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table API
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>
> The scope of this sub task includes:
> # Partial aggregate interface.
> # Simple aggregate function implementation, such as SUM/AVG/COUNT/MIN/MAX.
> # DataSetAggregateRule which translate logical calcite aggregate node to Flink user functions. As hash-based combiner is not available yet(see PR #1517), we would use sort-based combine as default.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)