Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGLgxbYpooagxMxwxHRBNC0U1JJFJ7CC17T7fh91kh+mY7xXgw@mail.gmail.com>
References: <ec788a10-3dcd-2747-a84d-23a1621e8d55@cern.ch> <CAGLgxbYpooagxMxwxHRBNC0U1JJFJ7CC17T7fh91kh+mY7xXgw@mail.gmail.com>
From: Stephan Ewen <sewen@apache.org>
Date: Fri, 12 Aug 2016 10:54:58 +0200
Message-ID: <CANC1h_s5wn9jL1mjGVsqrx1bQqN_-R89nhgKyvUUuQP45vJ3nw@mail.gmail.com>
Subject: Re: Conceptual difference Windows and DataSet
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a113f5fac8424610539dc09bb
archived-at: Fri, 12 Aug 2016 09:14:46 -0000

--001a113f5fac8424610539dc09bb
Content-Type: text/plain; charset=UTF-8

Hi Kevin!

The windows in Flink's DataStream API are organized by key. The reason is
that the windows are very flexible, and each key can form different windows
than the other (think sessions per user - each session starts and stops
differently).

There has been discussion about introducing something like "aligned
windows". These types of windows would be the same across all keys and
could therefor be globally organized. One could even think that these offer
DataSet-like features.
That is a bit into the future, still.

Greeting,
Stephan


On Sat, Aug 6, 2016 at 11:58 PM, Theodore Vasiloudis <
theodoros.vasiloudis@gmail.com> wrote:

> Hello Kevin,
>
> I'm not very familiar with the stream API, but I think you can achieve what
> you want by mapping over your elements to turn the
> strings into one-item lists, so that you get a key-value that is (K:
> String, V: (List[String], Int))  and then apply the window reduce function,
> which produces a data stream out of
> a windowed stream, you combine your lists there and sum the value. Again,
> it's not a great way to use reduce, since you are growing the list with
> each reduction.
>
> Regards,
> Theodore
>
> On Thu, Aug 4, 2016 at 1:36 AM, Kevin Jacobs <kevin.jacobs@cern.ch> wrote:
>
> > Hi,
> >
> > I have the following use case:
> >
> >     1. Group by a specific field.
> >
> >     2. Get a list of all messages belonging to the group.
> >
> >     3. Count the number of records in the group.
> >
> > With the use of DataSets, it is fairly easy to do this (see
> > http://stackoverflow.com/questions/38745446/apache-flink-
> > sum-and-keep-grouped/38747685#38747685):
> >
> > |fromElements(("a-b", "data1", 1), ("a-c", "data2", 1), ("a-b", "data3",
> > 1)). groupBy(0). reduceGroup { (it: Iterator[(String, String, Int)], out:
> > Collector[(String, List[String], Int)]) => { val group = it.toList if
> > (group.length > 0) out.collect((group(0)._1, group.map(_._2),
> > group.map(_._3).sum)) } |
> >
> > So, now I am moving to DataStreams (since the input is really a
> > DataStream). From my perspective, a Window should provide the same
> > functionality as a DataSet. This would easify the process a lot:
> >
> >     1. Window the elements.
> >
> >     2. Apply the same operations as before.
> >
> > Is there a way in Flink to do so? Otherwise, I would like to think of a
> > solution to this problem.
> >
> > Regards,
> > Kevin
> >
>

--001a113f5fac8424610539dc09bb--