Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Mon, 6 Feb 2017 17:19:41 +0000 (UTC)
From: "Fabian Hueske (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13035976.1484768856000.7421.1486401581902@Atlassian.JIRA>
In-Reply-To: <JIRA.13035976.1484768856000@Atlassian.JIRA>
References: <JIRA.13035976.1484768856000@Atlassian.JIRA> <JIRA.13035976.1484768856549@jira-lw-us.apache.org>
Subject: [jira] [Commented] (FLINK-5564) User Defined Aggregates
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 06 Feb 2017 17:19:55 -0000


    [ https://issues.apache.org/jira/browse/FLINK-5564?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1585=
4402#comment-15854402 ]=20

Fabian Hueske commented on FLINK-5564:
--------------------------------------

Hi [~Shaoxuan Wang], I think it should be possible to split the first three=
 steps as follows:

1. Add the new UDAGG interface and migrate existing aggregation functions t=
o it.
We keep the current implementation and just add code and unit tests for the=
 implementations of the aggregation functions that implement the new interf=
ace.
2. Use the new aggregation function for batch tables.
We switch the implementation for the DataSet runtime code. We still keep th=
e old aggregation functions for the streaming code.
3. Use the new aggregation function for streaming tables.
We switch the implementation for the DataStream runtime code. In this step =
we remove the old aggregation functions and clean up.

Adressing 1, 2, and 3 in a single issue will result in a huge PR which will=
 be hard to review. I'd prefer several smaller steps with well defined scop=
e.

Regarding the discussion of the window OVER functions. It would be great if=
 you could post your comment (with which I agree) to the corresponding JIRA=
 issue and the discussion on the dev list otherwise it might not be noticed=
.

Thanks, Fabian


> User Defined Aggregates
> -----------------------
>
>                 Key: FLINK-5564
>                 URL: https://issues.apache.org/jira/browse/FLINK-5564
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table API & SQL
>            Reporter: Shaoxuan Wang
>            Assignee: Shaoxuan Wang
>
> User-defined aggregates would be a great addition to the Table API / SQL.
> The current aggregate interface is not well suited for the external users=
.  This issue proposes to redesign the aggregate such that we can expose an=
 better external UDAGG interface to the users. The detailed design proposal=
 can be found here: https://docs.google.com/document/d/19JXK8jLIi8IqV9yf7hO=
s_Oz67yXOypY7Uh5gIOK2r-U/edit
> Motivation:
> 1. The current aggregate interface is not very concise to the users. One =
needs to know the design details of the intermediate Row buffer before impl=
ements an Aggregate. Seven functions are needed even for a simple Count agg=
regate.
> 2. Another limitation of current aggregate function is that it can only b=
e applied on one single column. There are many scenarios which require the =
aggregate function taking multiple columns as the inputs.
> 3. =E2=80=9CRetraction=E2=80=9D is not considered and covered in the curr=
ent Aggregate.
> 4. It might be very good to have a local/global aggregate query plan opti=
mization, which is very promising to optimize UDAGG performance in some sce=
narios.
> Proposed Changes:
> 1. Implement an aggregate dataStream API (Done by [FLINK-5582|https://iss=
ues.apache.org/jira/browse/FLINK-5582])
> 2. Update all the existing aggregates to use the new aggregate dataStream=
 API
> 3. Provide a better User-Defined Aggregate interface
> 4. Add retraction support
> 5. Add local/global aggregate


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)