flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Introduction
Date Sat, 01 Nov 2014 19:50:31 GMT
Hi Manu!

I think the change to explicitly expose the group reduce key is separate
from the overhaul of the aggregation functions.

I have added a comment to the issue, outlining the ideas gathered to far.

There is also an interesting follow-up possible: Reduce on individual
fields. Frequently, the reduce affects only one field, as for example in
Spark's reduceByKey, which is a special case for key-value pairs. We could
add a nice generic version "reduceField(fieldExpression, ReduceFunction<f>,
)".

An example for how this may look in a program is:

DataSet<MyType> pairs = ...
pairs.groupBy("name")
        . reduceField("counts", (a, b) -> a + b);


Stephan




On Sat, Nov 1, 2014 at 12:01 PM, Manu Kaul <manohar.kaul@gmail.com> wrote:

> Hi All,
> I shall be joining DIMA as a Senior Researcher from
> Tuesday 4 Nov onwards.
>
> I have more of a background in C++ (for OO) and languages like
> SML and Haskell (for FP). So I am learning the Scala syntax at
> the moment.
>
> I came across the following improvement request:
> https://issues.apache.org/jira/browse/FLINK-553
>
> Seeing that I have not yet had a chance to look at the Scala API
> and internals of Flink, would it be ok for me to work on this improvement
> or is this part of a bigger change? I see that most of the aggregation
> functions will anyway need a major overhaul, so I am not sure if this
> change also falls under the same ambit?
>
> I think the idea might be to return more data from the first function so
> that the next function in line in the composition has more to take as
> input. I am not sure if there are many users already using Flink
> because such a change would obviously break a lot of code.
>
> If this issue has other dependencies that I am not seeing at the
> moment, then please feel free to suggest some other Scala related
> change.
>
> Thanks,
> Manu
>
>
> --
>
> The greater danger for most of us lies not in setting our aim too high and
> falling short; but in setting our aim too low, and achieving our mark.
> - Michelangelo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message